diff --git a/CHANGELOG.md b/CHANGELOG.md index f2ee5f0..7db3bd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,1305 +1,139 @@ # Changelog -What has actually landed in ipodderx-rs, newest first. Each entry says what changed and, where it -matters, what was wrong before -- the reasoning is the point, not the diff. - -See [README.md](README.md) for what the thing is, and [docs/](docs/) for how to run it. - -## 2026-09-11 — Import an OPML file by uploading it - -The import screen now has a file picker as well as the paste box. The page reads the file and -checks it looks like OPML before sending anything. If it doesn't, the page says so and clears the -picker, and it also clears the picker after an import. The file travels as text and is never -written to disk on the server, so there is nothing to clean up there. - -The server now parses the OPML before touching anything, and returns `400` "that is not an OPML -file" instead of a `500`. `subscribe_opml` takes a parsed document, so `ipx import` also refuses a -non-OPML file by name before changing anything. - ---- - -## 2026-09-11 — OPML import and export are per person - -Importing an OPML now subscribes you to every feed in it. Feeds already in the catalogue cost -nothing, and unknown ones are added under the OPML's title. Before this, import only added URLs -missing from `config.toml` and subscribed nobody. So importing an export from another account did -nothing at all, and a genuinely new feed had no subscriber and was never scanned. The page said -"Imported 0 feed(s)". `ipx import` had the same gap; it now subscribes the first admin. Both go -through `subscribe_opml`. - -Export now lists only your own subscriptions. It used to write out the whole catalogue to anyone -signed in, including other people's feeds and any private URLs in them. - -Tests now cover every way in and out: -- Settings' Export OPML download and paste-to-import screen. -- A round trip that imports the admin's actual export into a second account and checks both - exports match. That round trip is exactly what failed. -- `ipx import` and `ipx export`, run in their own scratch config and database. - ---- - -## 2026-09-11 — The log is admin-only - -`GET /api/logs` now returns `403` to anyone who is not an admin, and the page hides the Log button -from them. Before this, every signed-in person could read the whole log. That includes every -account's name, every feed anyone subscribes to, and every failed sign-in. `/api/events` stays open -to everyone, because it carries the scan progress each person's page shows. - ---- - -## 2026-09-11 — Managing users from the web - -Settings has a **Manage users…** screen for an admin. From it you can add someone, with a password -or with none for someone the proxy signs in, tick or untick Admin, and remove an account. It is -backed by `GET/POST /api/users` and `PATCH/DELETE /api/users/{id}`, which return `403` for anyone -who is not an admin. The only admin cannot be demoted or removed, because nobody would then be -able to manage accounts except from the CLI on the box. Before this, accounts could only be managed -with `ipx user`. - ---- - -## 2026-09-11 — Unread feeds first inside an OPML - -An OPML subscription's feeds, both in the sidebar folder and on its own page, now list the ones with -unread items first. They were listed alphabetically, so with dozens of feeds the few with anything -new were scattered through the list. Within each half the order is still alphabetical. The browser -suite's fixture OPML gained a second feed, Aardvark Radio, which sorts first by name and by position, -so the new test only passes if unread wins. - ---- - -## 2026-09-11 — Deploying is a Docker image - -Production moved from a hand-started daemon in code-server to the `iPodderX` container in the Arcane -project `content`. `CLAUDE.md` now deploys by pushing to the registry at `192.168.1.130:5000` and -recreating that one service with `docker compose`. The old instructions copied a binary over a -process nobody supervised, so it did not come back after a reboot. Two known gaps are gone: the -image does support accounts, and the entrypoint drops to `99:100`, so downloads are no longer -owned by root. The README's Docker section said `docker compose up -d` builds the image and named the -service `ipx`; the compose file pulls from the registry and the service is `ipodderx`. - ---- - -## 2026-09-11 — Documentation - -`PROGRESS.md` became this changelog; the finished step lists moved to an appendix. The README is now -an overview that points at `docs/`: [configuration](docs/configuration.md) (every key, path and -environment variable), [cli](docs/cli.md), [users](docs/users.md) (what several people share), -[sso](docs/sso.md) (refreshed for accounts and the admin-only settings), and -[architecture](docs/architecture.md) (modules, the scan, the schema, the socket protocol, the HTTP -API). `CLAUDE.md` collects what someone working on the code needs to know and has already been -learned the hard way here -- `pkill -x` not `-f`, the page being compiled in, the dead columns on -`entries`, the Playwright worker that deleted its own database. - ---- - -## 2026-09-11 — Pruning respects a star from anyone - -Same defect as the reaper, one function along: `prune_entries` guarded on `entries.flagged`, which -nothing has written since read state moved to `entry_state`. Starring a text item with no file would -not have saved it from the age sweep. It follows the reaper's rule now, and takes orphaned read -state with whatever it deletes. - ---- - -## 2026-09-11 — Step D: one file, and everyone who wants it - -The last of it, which is all about telling the truth before acting: - -* **Delete says whose file it is.** A feed with other subscribers labels the button *Delete for - everyone*, and the confirmation names how many people share it. The server then has the last - word: if anyone else has starred the item or not played it yet, `DELETE /api/enclosures/{id}` - returns **409** with the reason in plain words, and only `?force=true` goes through. So the - rule is enforced where it matters rather than in the page that happens to be asking. -* **A feed's header says it is shared** -- "shared with 1 other person" -- which is the answer to - "why is there a file here I never asked for": someone else's subscription fetched it, and one - copy serves you both. -* Retention already respects the same rule from the entry before this one: starred by anyone keeps - it, read by everyone releases it. - -`others_wanting` is tested with three subscribers disagreeing, and a browser test walks the whole -delete flow: the label, both prompts, declining the second, and the file still being there. - ---- - -## 2026-09-11 — Retention caught up with per-user state - -Moving read and starred into `entry_state` left `reap_candidates` reading `entries.read` and -`entries.flagged`, which nothing writes any more: **starring stopped protecting a file**, and the -"delete the ones already read first" ordering was dead. Harmless while the quota and age limits are -0 -- the reaper deletes nothing at all then -- but it would have bitten the moment one was set. - -One file serves every subscriber, so both tests are now about all of them: **anyone** starring it -keeps it, and it only counts as read once **everyone** subscribed has read it. A file whose feed -nobody subscribes to has nobody left to keep it, so it sorts with the read ones. Tested with two -subscribers disagreeing. - ---- - -## 2026-09-11 — Steps B and C: what is yours, what is everyone's - -Read, starred and playback position moved out of `entries` into `entry_state (user_id, feed_id, -guid, ...)`; subscriptions became rows in `subscriptions (user_id, feed_id, ...)` carrying **your** -keywords, auto-download, explicit and per-scan limit. The feed list, unread counts, filters and -mark-all-read are all per person now. On first start the existing library is adopted by the admin: -2438 read/starred items and all 86 feeds, so nothing was lost. - -The split follows from the file being shared: - -* **Yours**: read state, starred, position, keywords, auto-download, explicit, per-scan limit, - and which feeds you see at all. -* **Everyone's**: the feed URL, its download folder, and when it is scanned -- there is one copy of - a file however many people subscribe, so those describe the file, not a preference. Admin-only, - refused with a 403 for anyone else rather than merely hidden. - -Scanning merges the subscribers' wants, because one fetch and one file serve them all: an item is -downloaded if **anyone** wants it (any one person's keyword set matching is enough, and one person -taking everything removes the filter), auto-download is on if anyone has it on, and the per-scan cap -is the largest anyone asked for. `merge_policy` is a pure function with a test covering each of -those. Subscribing to a feed someone already has costs no second fetch and no second copy on disk; -unsubscribing takes it off your list alone, and only when the last subscriber leaves does the feed -stop being scanned. - -**A test-harness bug worth naming**: Playwright imports the config in every worker, so the fixture's -`prepare()` ran again mid-run and deleted the data directory out from under the daemon. The daemon -kept serving from the unlinked inode while the CLI and any query opened a fresh empty database at -the same path -- which looked exactly like sign-in being broken. Only the launching process wipes -now (a worker has `TEST_WORKER_INDEX`). - ---- - -## 2026-09-10 — Scanning is the operator's decision - -The per-feed **Check schedule** picker is gone from feed settings, and the global Settings page is -hidden from anyone who is not an admin. Both are enforced in the handlers rather than merely hidden: -`PATCH /api/settings` and a `schedule` in `PATCH /api/feeds/{id}` return 403 for an ordinary user. -Polling costs bandwidth, is what a publisher notices, and one impatient setting affects everyone -reading that feed -- it belongs in config.toml. - -Folders, keywords, per-feed download limits and the feed URL stay editable by anyone signed in. - -`docs/sso.md` covers putting Cloudflare Zero Trust or Authentik in front of ipx: the tunnel and -Access application, the Authentik proxy provider and its forward-auth nginx block, the three lines -of ipx config each needs, and why `trusted_proxies` must name the proxy rather than a subnet -- -with the command to prove the refusal works. - ---- - -## 2026-09-10 — Marking an item read - -Two bugs in one place. `epAction`'s `redraw` closure called *itself* when it had a row to -update -- `if(el) redraw()` where it meant to swap the row -- so the Mark read button in the -text pane recursed until the stack blew. It now swaps that one row in place and refreshes the -text below only when it is the item being read. - -And opening an item now marks it read, which is what clicking a thing to read it means. The row -is redrawn where it stands rather than the list reloaded, so an item does not vanish from under -the pointer on the Unread tab. - ---- - -## 2026-09-10 — Step A: accounts and sign-in - -`users` and `sessions` tables, Argon2id hashing, a session cookie, and `ipx user add|list|passwd|rm` -(passwords come in on stdin, so they miss the shell history and any `ps` listing). - -Three ways in, in order of how specific the claim is: - -1. **A proxy header** naming the user -- `Cf-Access-Authenticated-User-Email` for the Cloudflare - Zero Trust in front of `ipodderx.sdf1.net`. Honoured **only** from an address in - `trusted_proxies` (loopback by default): a header is worth exactly as much as the hop that set - it, and the LAN port would otherwise let anyone claim to be anyone. Verified both ways -- a - spoof from an untrusted address is refused. -2. **A session cookie** from signing in at `/login`. -3. **The shared token**, which is the admin, so the healthcheck and existing links keep working. - -A database with no accounts creates **admin / ipodderx** and says so loudly in the log. The UI shows -who is signed in above the sidebar footer, with a sign-out, and a 401 sends the page to `/login`. - -Nothing is per-user *yet*: everyone still sees the same feeds and read state. That is step B. - ---- -## 2026-09-10 — Items, not episodes - -Half the library is text feeds, so the UI no longer calls everything an episode: counts, the search -box, the empty detail pane, the phone back button, the retention and per-feed settings, and the -download dialog all say **item** now. The sidebar's `20 eps` reads `20 items`. - -`S1E1` badges stay -- those come from `itunes:episode`/`itunes:season` and only appear when a feed -actually publishes them -- as does the `episode` column, which is that same field. - ---- - -## 2026-09-10 — Scanning stopped strobing - -A scan of 85 feeds fires 85 `feed_done` events, and the UI refreshed on every one: a full sidebar -rebuild plus an episode reload each time, which read as flicker. Bursts now collapse into one -refresh (500ms trailing), the per-feed "N new" toasts add up into a single summary, and both lists -keep their scroll position across a rebuild instead of jumping to the top. - -Measured over a full scan: 11 feed refreshes and 2 episode reloads in 45 seconds, down from one per -event. - -Settings and Log moved out of the header and toolbar into a footer under the feed list -- they are -housekeeping, not daily controls. The theme toggle stays in the header. - ---- - -## 2026-09-10 — Defaults and chrome - -* **All** is the default episode filter and the first tab; Unread, Downloaded and Flagged follow. - Opening a feed and seeing nothing because everything in it was read is a poor first impression. -* **OPML import/export moved into Settings**, under a Subscriptions heading, and the OPML button is - gone from the sidebar -- it is a thing you do once in a while, not a daily control. -* **The feed actions are pills with icons**, and Unsubscribe is pushed to the far end in a quiet - style: it sat next to Settings looking exactly like it, one slip away from losing a feed. - ---- - -## 2026-09-10 — Phone layout - -The UI was unusable on a phone, starting with the worst of it: the ☰ button lived inside the player -bar, which is hidden until something plays, so there was no way to reach the feed list at all. It -now sits in a small bar at the top of the main pane that is always there, and the sidebar slides -over the page with a scrim to tap away. - -The rest: - -* **One pane at a time.** The three-pane split becomes a list, with the item text taking the whole - screen over it and a `← Episodes` button back. The divider is hidden. -* **No sideways scrolling.** A grid column is min-content wide by default, so one long headline - ("davewiner/hackerNewsStars") dragged the entire page off the right edge -- `min-width:0` down the - shell/main/wrap chain, `overflow-wrap:anywhere` on headings, and an explicit - `minmax(0,1fr)` column for the OPML child list. -* **The player stacks**: title row on top, transport and seek bar below, and it paints above the - reading pane so it stays reachable. -* Header artwork, buttons and log rows shrink to fit; tap targets go to 38px. - -A Playwright case at 390x844 locks in the three things that actually broke: the burger is visible -with nothing playing, the page does not scroll sideways, and an item opens and closes over the list. - ---- - -## 2026-09-10 — Marking a subscription read - -An OPML subscription's page now has **Mark all read**, sitting where every other feed keeps it -- -before Unsubscribe. The fix is in the handler rather than the button: `read-all` resolves the feeds -whose group is the given id (through `subscriptions()`, so a child promoted to config counts too) -and marks those, since the subscription's own row holds no entries and marking it did nothing. - ---- - -## 2026-09-10 — A folder counts what it holds - -An OPML subscription has no entries of its own, so its row always read `0 unread` no matter how much -was waiting inside it. Unread and saved are now summed from the feeds it holds -- from all of them, -not just the ones a search filter left showing, so the number doesn't move as you type. The badge -caps at `999+` (the real figure is in its tooltip); a four-digit count ate the title beside it. - ---- - -## 2026-09-10 — Sidebar alignment - -The feed list had four different left edges: a row with no children skipped the chevron entirely, so -its artwork sat a chevron-width left of a folder's; children then used a different indent *and* a -smaller icon. Nothing lined up with anything. - -Every row now reserves the chevron slot whether or not it opens (an empty one is -`pointer-events:none`, so the click falls through to the row), all artwork is one size, and nesting -reads from the indent alone. Both label lines are `display:block` on a shared line-height instead of -an inline baseline, and the unread count has a `min-width` so three-digit feeds don't shove the -title. Rows came out shorter, so more feeds fit without scrolling. - ---- - -## 2026-09-10 — An item's picture - -An item's artwork now resolves in order of how deliberate the source is: `itunes:image`, then Media -RSS `media:thumbnail`, then a `media:content` that declares itself an image, and finally an image -**enclosure**. That last one matters here -- Substack puts each article's header picture in an -``, which is why those blog entries had no artwork despite carrying one all along. Audio -enclosures are never mistaken for pictures. - -Backfilling needed the validators cleared first: `record_entry` fills a missing image on update, but -a 304 skips parsing entirely, so the feeds would have kept their blank squares. (The self-heal added -earlier only fires when a feed has *zero* entries, which was not the case here.) - -Result across the library: 325 of 3470 entries now carry their own picture, 13 feeds where every -entry has one, 69 feeds that publish no per-item image at all -- those fall back to the feed's -artwork, which is the intended behaviour rather than a gap. - ---- - -## 2026-09-10 — Database cleanup, and the 304 trap it walked into - -Cleaned up on request: removed the `CT Log Archive Torrents` folder (123 preallocated files from the -abandoned 1.6 TB tuscolo torrent, **19 GB real** on disk, no row referencing any of it, feed no -longer subscribed), the unsubscribed ct-log feed's rows, and every OPML-derived entry/enclosure that -held no file so a rescan could rebuild them with the current parser. Kept every row holding a file, -so nothing on disk was orphaned, and kept the skipped/reaped history, without which the next scan -would re-download the 149 images just deleted. Disk 22 GB -> 3.8 GB, database 22.8 MB -> 5.2 MB. - -Correction worth recording: mid-download I checked that torrent folder, saw `du -sh` report 8 KB, -and told Ray it was sparse with nothing written. By the time it was abandoned it had allocated 19 GB. -The reassurance had a shelf life I did not mention. - -**Then "Abort Retry Fail is empty".** The cleanup deleted entries but left each feed's -ETag/Last-Modified. The rescan sent them, servers answered 304 (120 times in the log), the daemon -skipped parsing, and 57 feeds stayed empty -- and would have until a publisher happened to change -something. Cleared the validators on the empty feeds and rescanned: entries 1524 -> 3009, feeds with -content 19 -> 64, Abort Retry Fail back to 20. - -Fixed in code so it cannot recur: a 304 arriving while the feed has **zero stored entries** means the -validator has outlived the data -- a restore, a manual edit, a cleanup. The daemon now believes the -database over the validator, drops it and asks again. Proved live by deleting a feed's entries, -leaving its ETag, and rescanning: 20 entries rebuilt, self-heal logged once. - -**"not a wanted media type"** was internal jargon reaching the UI, and it was stored in `last_error` -so an ordinary filter decision rendered in red as though something had failed. Reworded to "not -audio or video", 144 existing rows updated, and the UI now paints a reason red only when the state -is actually `error`. - ---- - -## 2026-09-10 — Multiple enclosures per item, and viewing without downloading - -**View without downloading.** A non-audio/video enclosure now carries a View link opening in a new -tab: the publisher's own URL when nothing is downloaded, the local copy at `/media/` when it is. -Deliberately a direct link rather than a proxy — relaying arbitrary URLs through the daemon would -make it a fetch-anything service. - -**Multiple enclosures.** Probed before assuming, and the result was worse than expected: the `rss` -crate models an item as having at most one enclosure (which is what RSS 2.0 says) and when a feed -carries several it silently keeps **the last**, dropping the rest. So a two-file item lost its first -file entirely. - -`enclosures_by_item()` reads them straight from the XML with quick-xml, in document order, -unescaping attribute values — a feed URL's `&` arrives as `&`, so skipping that would corrupt -every query string. It falls back to the parsed enclosure if the scan and the parser disagree on -item count. Atom already collected all `rel="enclosure"` links. The row now summarises the enclosure -you would act on (playable, else downloaded, else first) and says "+N more files"; the pane below -lists them all. - -**A real limitation surfaced by a broken fixture.** Two browser tests failed with zero enclosures in -the detail pane. Not the new scanner — verified by running it against the fixture files directly, -which was right every time. The cause: my fixtures pointed two feeds at the *same* enclosure URL, -and `enclosures.url` is UNIQUE across the whole database, so whichever feed is scanned first claims -it and the other's entry gets nothing. That is the dedupe working as designed, but it means **two -feeds legitimately sharing a media URL will only ever show it under one of them** — worth knowing, -and worth revisiting if a network feed and a show feed ever overlap. - ---- - -## 2026-09-10 — A file is not the same as a playable file - -Reported: "Abort Retry Fail still shows downloaded and audio playback UI". The media-type filter -stopped *new* image enclosures being fetched, but those 20 JPEGs were already on disk from before -it existed — and the UI gave a play button and an `