rays/ipodderx-rs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Documentation: docs/, a changelog, and CLAUDE.md

Browse Source 
PROGRESS.md becomes CHANGELOG.md with the finished step lists moved to an
appendix. The README is an overview pointing at docs/: configuration,
cli, users, sso (refreshed for accounts and admin-only settings), and
architecture. CLAUDE.md collects what working on this code actually
requires -- pkill -x not -f, the page being compiled in, the dead columns
on entries, the Playwright worker that deleted its own database.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AdXho5tTkjFLeUXKbEjKBh
This commit is contained in:
Ray Slakinski
2026-09-11 03:00:25 +00:00
parent c47c224372
commit 6114add4a6
8 changed files with 684 additions and 269 deletions
Download Patch File Download Diff File Expand all files Collapse all files

134
docs/architecture.md Normal file
View File

@@ -0,0 +1,134 @@
# How it works
One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a Unix-socket
control server, and the web UI. Everything else is a CLI that either does the work itself or hands
it to a running daemon.
## Modules
| File | Responsibility | What it replaced in the Python |
|---|---|---|
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
| `src/db.rs` | SQLite schema, migrations, every query | `.ipxd` plists, `history.dat`, `qmcache.dat` |
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
| `src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` |
| `src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout |
| `src/auth.rs` | Argon2id hashing, session tokens, header names | — |
| `src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — |
| `src/logbuf.rs` | Ring buffer behind the UI's Log view | — |
| `web/index.html` | The whole front end, `include_str!`d into the binary | — |
The page is compiled in, so **editing `web/index.html` needs a rebuild**.
## A scan
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
re-derived into the database each scan, never written to config.toml.
4. Record entries. A changed title or description flips the item back to unread.
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
a reason.
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
or an error page — and is deleted.
## Data model
```
feeds id, url, title, image, etag, last_modified, last_checked, ttl_mins,
last_error, orphaned, group_id, managed
entries feed_id, guid, title, link, published, description, first_seen,
image, duration, episode, season PK (feed_id, guid)
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,
bytes_done, downloaded_at, last_error
users id, name, pass_hash, is_admin, created
sessions token, user_id, created, seen
subscriptions user_id, feed_id, keywords, auto_download, allow_explicit,
max_new_per_check, created PK (user_id, feed_id)
entry_state user_id, feed_id, guid, read, flagged, position
PK (user_id, feed_id, guid)
```
`entries` still has `read`, `flagged` and `position` columns from before accounts existed. They are
**dead** — the migration copied them into `entry_state` and nothing reads them now. Anything found
querying them is a bug; two were.
Schema changes: add the table or column to `SCHEMA`, and for a column also to the list in
`migrate()`, which does `PRAGMA table_info` then `ALTER TABLE ADD COLUMN`. `Db::memory()` runs the
same path as `Db::open`, so a migration-only column cannot pass tests while missing in production.
## Control socket
Newline-delimited JSON, both directions, over `$XDG_RUNTIME_DIR/ipx.sock`.
```sh
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
{"ev":"feed_start","feed":"atp"}
{"ev":"progress","feed":"atp","enclosure":42,"file":"ep1.mp3","done":8192,"total":3000000}
{"ev":"download_done","feed":"atp","enclosure":42,"path":"…","bytes":3000000}
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
{"ev":"scan_done","feeds":1}
```
**Commands** — `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `download`
(`enclosure`), `status`.
**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
there.
Progress carries the enclosure id, without which a UI cannot tell one download from another and
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
so a client attached to a busy daemon also sees that daemon's other work.
Inside the process the same events go over a `tokio::broadcast`; commands arrive on an `mpsc` and
are handled by a single worker, so nothing races over the same download. Shutdown is a `watch`
channel raced *inside* each job — `tokio::select!` only races branches at the point of selection,
so a long download had to be able to notice the signal itself.
## HTTP API
Everything below `/api` needs a signed-in user; the browser gets a redirect to `/login`, anything
else a `401`.
| Route | |
|---|---|
| `GET /` | the app |
| `GET /login`, `POST /api/login`, `POST /api/logout`, `GET /api/me` | sign-in |
| `GET /api/feeds`, `POST /api/feeds` | your subscriptions; subscribe |
| `PATCH /api/feeds/{id}`, `DELETE /api/feeds/{id}` | your settings or (admin) the feed's; unsubscribe |
| `GET /api/feeds/{id}/entries` | paged, filtered, searchable |
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, starred, position |
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
| `POST /api/fetch`, `GET /api/opml`, `POST /api/opml` | |
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
| `GET /api/events` | SSE, the same broadcast the socket carries |
| `GET /api/logs` | the ring buffer, with a sequence cursor |
| `GET /media/{id}` | the file, with Range support so seeking works |
Show notes are feed-supplied HTML from an untrusted source, sanitized with `ammonia` server-side
before they reach the page.
## Testing
```sh
cargo test # parsing, filters, retention, schedules, SQL, per-user isolation
node tests/page-smoke.js # the page script loads and every selector it wires at load exists
npx playwright test # a real browser against a real daemon on fixture feeds
```
The Rust tests cannot see a wrong selector, a handler that runs and does nothing, or a page that
renders empty — which is what has actually reached users. Each Playwright case maps to a bug that
did.
The suite starts its own daemon and database under `/tmp/ipx-ui-test`, wiped once per run. Tests
share that daemon and run in order, so a test that marks something read changes what later tests
see — make assertions that do not depend on earlier ones.

91
docs/cli.md Normal file
View File

@@ -0,0 +1,91 @@
# Command line
```
ipx [--config PATH] [--local] <command>
```
Every command that has a wire form probes the control socket first: if a daemon is running, the
daemon does the work and the CLI just renders the events it streams back. That is deliberate — two
processes must never download the same thing. `--local` forces the work to happen in-process.
| Command | What it does |
|---|---|
| `ipx list` | Subscriptions and their state |
| `ipx status` | Counts: feeds, pending, downloaded |
| `ipx fetch [FEED] [--force]` | Scan everything, or one feed. `--force` ignores the TTL |
| `ipx add <url> [--folder X] [--keywords a,b]` | Subscribe; the id comes from the feed title |
| `ipx rm <feed>` | Unsubscribe; downloads and history are kept |
| `ipx import <file.opml>` / `ipx export <file.opml>` | Move subscriptions in or out |
| `ipx reap [--dry-run]` | Run retention now |
| `ipx user <add\|list\|passwd\|rm>` | Accounts for the web UI |
| `ipx daemon [--web ADDR]` | Scheduler, control socket and web UI |
## Accounts
Passwords are read from **stdin**, so they miss the shell history and any `ps` listing.
```sh
echo -n 'a good password' | ipx user add ray # local account
ipx user add ray@example.com --no-password # signs in through the proxy only
echo -n 'a good password' | ipx user passwd admin # change a password
ipx user list # who exists, and how each signs in
ipx user rm sam # account, subscriptions and read state
```
The first account created is an admin; later ones are ordinary users. A database with no accounts
at all gets **admin / ipodderx** on the next daemon start, announced in the log — change it.
To avoid even the command line, read it interactively:
```sh
read -s PW && echo -n "$PW" | ipx user passwd admin
```
## Scanning
```sh
ipx fetch # everything due
ipx fetch atp --force # one feed, ignoring its TTL and schedule
```
A scan: conditional GET (`If-None-Match` / `If-Modified-Since`), parse, record new entries, apply
the filters, then download up to the per-scan cap, newest first. A feed nothing has changed in
answers `304` and costs one request.
## Retention
```sh
ipx reap --dry-run # what would go, oldest first
ipx reap # actually delete
```
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
with no file are pruned from the database. **Starred by anyone keeps a file**, and one only counts
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
stops the next scan fetching it again.
## The daemon
```sh
ipx daemon # scheduler + socket + web UI
ipx daemon --web 0.0.0.0:8099 # override the configured bind for one run
```
One daemon per socket; a second refuses to start rather than fight over the database. It shuts down
cleanly on SIGTERM, including mid-download.
To kill it, match the binary exactly:
```sh
pkill -x ipx
```
`pkill -f ipx` matches the shell running the command too, and kills your own session.
## Talking to it directly
```sh
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
```
See [architecture.md](architecture.md#control-socket) for the protocol.

118
docs/configuration.md Normal file
View File

@@ -0,0 +1,118 @@
# Configuration
One TOML file, read at startup and re-read whenever the web UI writes to it — most changes take
effect without a restart. Default location `$XDG_CONFIG_HOME/ipx/config.toml`
(`~/.config/ipx/config.toml`), overridden with `--config` or `$IPX_CONFIG`.
| What | Where | Override |
|---|---|---|
| Config | `~/.config/ipx/config.toml` | `--config`, `$IPX_CONFIG` |
| Database | `~/.local/share/ipx/state.db` | `$IPX_DATA_DIR` |
| Control socket | `$XDG_RUNTIME_DIR/ipx.sock` | `[general] socket` |
| Downloads | `[general] download_dir` | — |
`~` is expanded in paths. The database is SQLite in WAL mode; back it up by copying `state.db`
while the daemon is stopped, or with `sqlite3 state.db .backup`.
## `[general]`
```toml
[general]
download_dir = "~/Podcasts"
socket = "/run/user/1000/ipx.sock"
schedule = "every 1h" # "every 30m", "every 4h", "2d", "90" (minutes)
organize = "feed" # "feed" | "date"
max_total_gb = 50 # 0 = unlimited
max_age_days = 30 # 0 = keep forever
max_new_per_check = 3 # per feed, per scan. 0 = unlimited
media_types = ["audio", "video"]
```
* **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
pad. Starred items are never deleted, and a file only counts as read once every subscriber has
read it. `0` disables it entirely.
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
Starred ones stay. `0` disables it.
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
fetched 216 files and 22 GB in one scan.
* **`media_types`** — top-level MIME types taken automatically. Anything else is still listed and
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
without this the disk fills with artwork. Empty takes everything.
`interval_mins` from older configs is still read, and `schedule` supersedes it.
## `[torrent]`
```toml
[torrent]
enabled = true
seed_ratio = 1.0 # stop seeding at this ratio ...
seed_time_mins = 60 # ... or after this long, whichever comes first
port_range = "6881-6889"
stall_mins = 30 # give up on a torrent making no progress
```
A `.torrent` body is handed to the torrent path whatever MIME type it was advertised as. Torrents
run on their own tasks (two at a time) so a slow swarm never blocks a scan.
## `[web]`
```toml
[web]
enabled = true
bind = "0.0.0.0:8099" # 127.0.0.1:8080 by default
token = "" # generated and saved on first run
trusted_header = "" # e.g. "Cf-Access-Authenticated-User-Email"
trusted_proxies = ["127.0.0.1", "::1"]
auto_create_users = true
session_days = 30
```
* **`token`** — the shared secret, which signs in as the **admin**. `?token=…` sets a cookie, so
you paste it once per browser. It is what the Docker healthcheck and any scripts use.
* **`trusted_header`** — a header naming the signed-in user, set by whatever fronts ipx. Empty
disables that path. See [sso.md](sso.md).
* **`trusted_proxies`** — addresses allowed to assert that header, and the entire security boundary
for it. Name the proxy, never a subnet.
* **`auto_create_users`** — create an account the first time the proxy vouches for a new name.
* **`session_days`** — sign a session out after this long without a request.
It is plain HTTP. On a LAN bind everything crosses the network in the clear — and a feed URL can
itself carry a credential. Put TLS in front of it if that matters.
## `[feeds.<id>]`
The table key is the feed id: stable, human-readable, and used in paths and the API. `ipx add`
derives it from the feed title.
```toml
[feeds.atp]
url = "https://atp.fm/rss"
folder = "Accidental Tech Podcast" # default: the feed title
schedule = "every 6h" # overrides [general] for this feed
media_types = ["audio"] # overrides [general] for this feed
username = "ray" # HTTP basic auth
password_env = "IPX_ATP_PASS" # preferred over a literal `password`
```
With more than one account, **`keywords`, `auto_download`, `allow_explicit` and
`max_new_per_check` live on each person's subscription in the database**, not here — the values in
config.toml are the fallback for a feed nobody has claimed. The keys above describe the feed itself
and are the same for everyone. See [users.md](users.md).
Feeds derived from a subscribed OPML are **not** written here: the OPML is the source of truth and
they are re-derived on every scan. Editing one in the UI promotes it to a real config entry.
## Environment
| Variable | Effect |
|---|---|
| `IPX_CONFIG` | Config file path |
| `IPX_DATA_DIR` | Directory holding `state.db` |
| `IPX_LOG` | What reaches stderr (`ipx=debug`, `ipx::scan=debug`, …) |
| `IPX_UI_LOG` | What the in-process log buffer captures for the UI's Log view |
| `http_proxy` / `https_proxy` | Honoured for feed and enclosure fetches |

26
docs/sso.md
View File

@@ -35,7 +35,15 @@ Restart the daemon after editing. Accounts made this way have **no password**: t
arrive through the proxy. `ipx user list` marks them `proxy only`.
The first account created is an admin. Every later one is an ordinary user, and an ordinary user
cannot change global settings or how often feeds are scanned. Promote someone with:
cannot change global settings, a feed's URL or folder, or how often feeds are scanned: the API
refuses those with a `403`, not just the UI. Everything else about a feed (which items they want,
whether to fetch them, how many at a time) is theirs alone; see [users.md](users.md).
Somebody arriving through the proxy for the first time starts with **no feeds**, because
subscriptions are per person. Adding a feed someone else already reads costs no second fetch and no
second copy on disk.
Promote someone with:
```sh
ipx user list
@@ -43,7 +51,9 @@ echo -n 'a good password' | ipx user passwd <name> # optional: also lets them
```
Local sign-in at `/login` keeps working alongside all of this, which is how you get in from the LAN
when the tunnel is down. A brand new database starts with **admin / ipodderx** — change it.
when the tunnel is down. So does the shared `[web] token`, which signs in as the admin: that is
what the Docker healthcheck uses, and the way back in if you lock yourself out. A brand new database
starts with **admin / ipodderx** — change it.
---
@@ -206,7 +216,13 @@ ipx user rm sam # remove the account
Set `auto_create_users = false` once everyone who should have an account has one. After that the
proxy vouching for an unknown name is logged and refused, rather than quietly making an account.
Pre-create people instead with `ipx user add <name> --no-password`, using exactly the name the
header will carry (Cloudflare sends the email address, lower-cased).
Scanning intervals, the disk quota, retention and the download folder are **admin-only** — the
Settings button is hidden for everyone else, and the API refuses the change even if the request is
made by hand. Ordinary users still control their own folders, keywords and downloads per feed.
Scanning intervals, the disk quota, retention, the download folder and a feed's URL are
**admin-only**: the Settings button is hidden for everyone else, and the API refuses the change even
if the request is made by hand. Everyone controls their own keywords, auto-download, explicit
setting and per-scan cap, along with their own read state and which feeds they see.
See also [users.md](users.md) for what several people share, [configuration.md](configuration.md)
for every `[web]` key, and [cli.md](cli.md) for the `ipx user` commands.

83
docs/users.md Normal file
View File

@@ -0,0 +1,83 @@
# Accounts, and what several people share
ipx serves any number of people from one copy of the data. The rule that decides everything else:
**there is one file on disk per enclosure URL.** Two people subscribed to the same show cost one
fetch, one parse and one file.
## What is yours, what is everyone's
| Yours alone | The same for everyone |
|---|---|
| Read, starred, playback position | The feed's URL |
| Which feeds you see at all | Its download folder |
| Keywords, auto-download, explicit, per-scan cap | When it is scanned |
| | The file on disk |
The right-hand column describes the feed and the file rather than a preference — two people wanting
different folders would mean two copies. Those three are **admin-only**, and the API returns `403`
for anyone else rather than merely hiding the controls.
## How the scanner merges everyone's wants
One fetch serves every subscriber, so the policy is a union:
* an item is downloaded if **anyone** wants it — one person's keyword set matching is enough, and
one person with no keywords removes the filter for that feed entirely
* auto-download is on if **anyone** has it on
* the per-scan cap is the **largest** anyone asked for
So "auto-download off" means *I don't cause downloads*, not *I never see them*. If someone else's
subscription pulls an item, you see it listed as downloaded and can play it, because the enclosure
is shared.
## Deleting
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
for everyone** and names them in the confirmation, and the server has the last word: if anyone else
has starred the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
reason, and only `?force=true` goes through.
Retention follows the same rule: starred by anyone keeps a file, and it counts as read only once
every subscriber has read it.
## Signing in
Three ways, tried in order of how specific the claim is:
1. **A proxy header** naming the user — Cloudflare Zero Trust or Authentik. Honoured only from an
address in `trusted_proxies`. See [sso.md](sso.md).
2. **A session cookie** from signing in at `/login`. Argon2id hashes, sessions in the database,
idle timeout `session_days`.
3. **The shared `[web] token`**, which signs in as the admin — this is what the Docker healthcheck
and any scripts use.
A database with no accounts creates **admin / ipodderx** on the next daemon start and says so in
the log. Change it:
```sh
echo -n 'a good password' | ipx user passwd admin
```
## Adding someone
```sh
echo -n 'their password' | ipx user add sam
```
They sign in at `/login` and start with **no feeds**: subscriptions are per person. Adding a feed
someone else already has costs nothing — no second fetch, no second copy — it just appears on their
list with their own read state. Unsubscribing removes it from their list alone; only when the last
subscriber leaves does the feed stop being scanned, and even then its files and history stay, so
re-subscribing does not pull the back catalogue again.
Accounts are managed from the command line only; there is no user administration in the web UI yet.
## Admin
The first account is an admin. An admin can change global settings (scanning interval, quota,
retention, media types, download folder) and a feed's URL, folder and schedule. Everyone else gets
the Settings button hidden and a `403` if they ask anyway.
```sh
ipx user list # the admin column says who
```