# How ipx was built The long-form record, newest first: what landed each day, what was wrong before, and what it cost to find out. [CHANGELOG.md](../CHANGELOG.md) has the short version, by release; this is where the reasoning lives. New write-ups go at the top. See [README.md](../README.md) for what the thing is. ## 2026-09-11 — The original's layout Ray pointed at a screenshot of the Mac app (techpp.com, 2012) and asked for its panes, its toolbar, and a Directory that lives in the feed list rather than behind a button. What it had, and what ipx now does: - A toolbar across the window: subscribe and unsubscribe, play, flag, refresh, and a search box scoped to the feed on show. ipx's has the same groups, acting on the selected feed and item, with Settings and Log at the right end, for admins only. - A source list opening with Directory, Playlist Builder and All Subscriptions above the feeds. ipx has Directory, Popular and All Subscriptions there, opening in the main pane. Playlist Builder is left out, since nothing here builds playlists. - The entries as a table, with a Files pane beside it and the entry below. The columns are unread, kept, the item, its feed, its file and when. Sorting by column is not done yet. - A status bar with the totals for what is on show. All Subscriptions needed one new endpoint, `GET /api/entries`. It is the per-feed query with `feed_id = ?` swapped for the person's subscriptions. The enclosure lookup that follows it used to filter by feed as well; a page can now span feeds, so each file is matched to its row by feed and guid instead. A phone has no room for a pane beside the table, so there the files follow the item's text in the full-screen reader, and the table drops to title and date. --- ## 2026-09-11 — Popular on this server The old iPodderX had a directory of podcasts and a top-feeds list. The open-sourced engine shows how the list was fed. After every successful download, `iPXClass.py:1249` called `doPing`, which sent the enclosure URL and the feed URL, unencoded and over plain HTTP, to `directory.iPodderX.com/feedData/survey/files`. It was on by default, and only the Mac app's preferences could switch it off (`anonFeedback`). The directory itself lived on that server and in the Cocoa app, and both are gone. ipx keeps the idea and drops the ping. Add feed lists what other accounts on this server subscribe to, ranked by subscriber count. The old ping would have sent a Patreon feed's `auth=` key along with everything else, so this is built around not doing that: - A row carries an id, a title, artwork and a count. No URL reaches the page, so no key can. - Subscribing goes by id through `POST /api/popular/{id}`, which checks the id against the same list. Guessing a private feed's id gets a 400. - Anything that looks private is never listed: a login configured for the feed, credentials in the URL, or a query key containing `auth`, `token`, `key`, `secret`, `pass`, `sig`, `session`, `user` or `uid`. It is a heuristic, and a token hidden in the URL's path gets through. A per-feed `unlisted` flag is the upgrade if that happens. - Feeds from an OPML are left out. Everyone subscribed to an OPML counts every feed inside it, so they would bury everything anyone chose on purpose. --- ## 2026-09-11 — Import an OPML file by uploading it The import screen now has a file picker as well as the paste box. The page reads the file and checks it looks like OPML before sending anything. If it doesn't, the page says so and clears the picker, and it also clears the picker after an import. The file travels as text and is never written to disk on the server, so there is nothing to clean up there. The server now parses the OPML before touching anything, and returns `400` "that is not an OPML file" instead of a `500`. `subscribe_opml` takes a parsed document, so `ipx import` also refuses a non-OPML file by name before changing anything. --- ## 2026-09-11 — OPML import and export are per person Importing an OPML now subscribes you to every feed in it. Feeds already in the catalogue cost nothing, and unknown ones are added under the OPML's title. Before this, import only added URLs missing from `config.toml` and subscribed nobody. So importing an export from another account did nothing at all, and a genuinely new feed had no subscriber and was never scanned. The page said "Imported 0 feed(s)". `ipx import` had the same gap; it now subscribes the first admin. Both go through `subscribe_opml`. Export now lists only your own subscriptions. It used to write out the whole catalogue to anyone signed in, including other people's feeds and any private URLs in them. Tests now cover every way in and out: - Settings' Export OPML download and paste-to-import screen. - A round trip that imports the admin's actual export into a second account and checks both exports match. That round trip is exactly what failed. - `ipx import` and `ipx export`, run in their own scratch config and database. --- ## 2026-09-11 — The log is admin-only `GET /api/logs` now returns `403` to anyone who is not an admin, and the page hides the Log button from them. Before this, every signed-in person could read the whole log. That includes every account's name, every feed anyone subscribes to, and every failed sign-in. `/api/events` stays open to everyone, because it carries the scan progress each person's page shows. --- ## 2026-09-11 — Managing users from the web Settings has a **Manage users…** screen for an admin. From it you can add someone, with a password or with none for someone the proxy signs in, tick or untick Admin, and remove an account. It is backed by `GET/POST /api/users` and `PATCH/DELETE /api/users/{id}`, which return `403` for anyone who is not an admin. The only admin cannot be demoted or removed, because nobody would then be able to manage accounts except from the CLI on the box. Before this, accounts could only be managed with `ipx user`. --- ## 2026-09-11 — Unread feeds first inside an OPML An OPML subscription's feeds, both in the sidebar folder and on its own page, now list the ones with unread items first. They were listed alphabetically, so with dozens of feeds the few with anything new were scattered through the list. Within each half the order is still alphabetical. The browser suite's fixture OPML gained a second feed, Aardvark Radio, which sorts first by name and by position, so the new test only passes if unread wins. --- ## 2026-09-11 — Deploying is a Docker image Production moved from a hand-started daemon in code-server to the `iPodderX` container in the Arcane project `content`. `CLAUDE.md` now deploys by pushing to the registry at `192.168.1.130:5000` and recreating that one service with `docker compose`. The old instructions copied a binary over a process nobody supervised, so it did not come back after a reboot. Two known gaps are gone: the image does support accounts, and the entrypoint drops to `99:100`, so downloads are no longer owned by root. The README's Docker section said `docker compose up -d` builds the image and named the service `ipx`; the compose file pulls from the registry and the service is `ipodderx`. --- ## 2026-09-11 — Documentation `PROGRESS.md` became this changelog; the finished step lists moved to an appendix. The README is now an overview that points at `docs/`: [configuration](configuration.md) (every key, path and environment variable), [cli](cli.md), [users](users.md) (what several people share), [sso](sso.md) (refreshed for accounts and the admin-only settings), and [architecture](architecture.md) (modules, the scan, the schema, the socket protocol, the HTTP API). `CLAUDE.md` collects what someone working on the code needs to know and has already been learned the hard way here -- `pkill -x` not `-f`, the page being compiled in, the dead columns on `entries`, the Playwright worker that deleted its own database. --- ## 2026-09-11 — Pruning respects a star from anyone Same defect as the reaper, one function along: `prune_entries` guarded on `entries.flagged`, which nothing has written since read state moved to `entry_state`. Starring a text item with no file would not have saved it from the age sweep. It follows the reaper's rule now, and takes orphaned read state with whatever it deletes. --- ## 2026-09-11 — Step D: one file, and everyone who wants it The last of it, which is all about telling the truth before acting: * **Delete says whose file it is.** A feed with other subscribers labels the button *Delete for everyone*, and the confirmation names how many people share it. The server then has the last word: if anyone else has starred the item or not played it yet, `DELETE /api/enclosures/{id}` returns **409** with the reason in plain words, and only `?force=true` goes through. So the rule is enforced where it matters rather than in the page that happens to be asking. * **A feed's header says it is shared** -- "shared with 1 other person" -- which is the answer to "why is there a file here I never asked for": someone else's subscription fetched it, and one copy serves you both. * Retention already respects the same rule from the entry before this one: starred by anyone keeps it, read by everyone releases it. `others_wanting` is tested with three subscribers disagreeing, and a browser test walks the whole delete flow: the label, both prompts, declining the second, and the file still being there. --- ## 2026-09-11 — Retention caught up with per-user state Moving read and starred into `entry_state` left `reap_candidates` reading `entries.read` and `entries.flagged`, which nothing writes any more: **starring stopped protecting a file**, and the "delete the ones already read first" ordering was dead. Harmless while the quota and age limits are 0 -- the reaper deletes nothing at all then -- but it would have bitten the moment one was set. One file serves every subscriber, so both tests are now about all of them: **anyone** starring it keeps it, and it only counts as read once **everyone** subscribed has read it. A file whose feed nobody subscribes to has nobody left to keep it, so it sorts with the read ones. Tested with two subscribers disagreeing. --- ## 2026-09-11 — Steps B and C: what is yours, what is everyone's Read, starred and playback position moved out of `entries` into `entry_state (user_id, feed_id, guid, ...)`; subscriptions became rows in `subscriptions (user_id, feed_id, ...)` carrying **your** keywords, auto-download, explicit and per-scan limit. The feed list, unread counts, filters and mark-all-read are all per person now. On first start the existing library is adopted by the admin: 2438 read/starred items and all 86 feeds, so nothing was lost. The split follows from the file being shared: * **Yours**: read state, starred, position, keywords, auto-download, explicit, per-scan limit, and which feeds you see at all. * **Everyone's**: the feed URL, its download folder, and when it is scanned -- there is one copy of a file however many people subscribe, so those describe the file, not a preference. Admin-only, refused with a 403 for anyone else rather than merely hidden. Scanning merges the subscribers' wants, because one fetch and one file serve them all: an item is downloaded if **anyone** wants it (any one person's keyword set matching is enough, and one person taking everything removes the filter), auto-download is on if anyone has it on, and the per-scan cap is the largest anyone asked for. `merge_policy` is a pure function with a test covering each of those. Subscribing to a feed someone already has costs no second fetch and no second copy on disk; unsubscribing takes it off your list alone, and only when the last subscriber leaves does the feed stop being scanned. **A test-harness bug worth naming**: Playwright imports the config in every worker, so the fixture's `prepare()` ran again mid-run and deleted the data directory out from under the daemon. The daemon kept serving from the unlinked inode while the CLI and any query opened a fresh empty database at the same path -- which looked exactly like sign-in being broken. Only the launching process wipes now (a worker has `TEST_WORKER_INDEX`). --- ## 2026-09-10 — Scanning is the operator's decision The per-feed **Check schedule** picker is gone from feed settings, and the global Settings page is hidden from anyone who is not an admin. Both are enforced in the handlers rather than merely hidden: `PATCH /api/settings` and a `schedule` in `PATCH /api/feeds/{id}` return 403 for an ordinary user. Polling costs bandwidth, is what a publisher notices, and one impatient setting affects everyone reading that feed -- it belongs in config.toml. Folders, keywords, per-feed download limits and the feed URL stay editable by anyone signed in. `docs/sso.md` covers putting Cloudflare Zero Trust or Authentik in front of ipx: the tunnel and Access application, the Authentik proxy provider and its forward-auth nginx block, the three lines of ipx config each needs, and why `trusted_proxies` must name the proxy rather than a subnet -- with the command to prove the refusal works. --- ## 2026-09-10 — Marking an item read Two bugs in one place. `epAction`'s `redraw` closure called *itself* when it had a row to update -- `if(el) redraw()` where it meant to swap the row -- so the Mark read button in the text pane recursed until the stack blew. It now swaps that one row in place and refreshes the text below only when it is the item being read. And opening an item now marks it read, which is what clicking a thing to read it means. The row is redrawn where it stands rather than the list reloaded, so an item does not vanish from under the pointer on the Unread tab. --- ## 2026-09-10 — Step A: accounts and sign-in `users` and `sessions` tables, Argon2id hashing, a session cookie, and `ipx user add|list|passwd|rm` (passwords come in on stdin, so they miss the shell history and any `ps` listing). Three ways in, in order of how specific the claim is: 1. **A proxy header** naming the user -- `Cf-Access-Authenticated-User-Email` for the Cloudflare Zero Trust in front of `ipodderx.sdf1.net`. Honoured **only** from an address in `trusted_proxies` (loopback by default): a header is worth exactly as much as the hop that set it, and the LAN port would otherwise let anyone claim to be anyone. Verified both ways -- a spoof from an untrusted address is refused. 2. **A session cookie** from signing in at `/login`. 3. **The shared token**, which is the admin, so the healthcheck and existing links keep working. A database with no accounts creates **admin / ipodderx** and says so loudly in the log. The UI shows who is signed in above the sidebar footer, with a sign-out, and a 401 sends the page to `/login`. Nothing is per-user *yet*: everyone still sees the same feeds and read state. That is step B. --- ## 2026-09-10 — Items, not episodes Half the library is text feeds, so the UI no longer calls everything an episode: counts, the search box, the empty detail pane, the phone back button, the retention and per-feed settings, and the download dialog all say **item** now. The sidebar's `20 eps` reads `20 items`. `S1E1` badges stay -- those come from `itunes:episode`/`itunes:season` and only appear when a feed actually publishes them -- as does the `episode` column, which is that same field. --- ## 2026-09-10 — Scanning stopped strobing A scan of 85 feeds fires 85 `feed_done` events, and the UI refreshed on every one: a full sidebar rebuild plus an episode reload each time, which read as flicker. Bursts now collapse into one refresh (500ms trailing), the per-feed "N new" toasts add up into a single summary, and both lists keep their scroll position across a rebuild instead of jumping to the top. Measured over a full scan: 11 feed refreshes and 2 episode reloads in 45 seconds, down from one per event. Settings and Log moved out of the header and toolbar into a footer under the feed list -- they are housekeeping, not daily controls. The theme toggle stays in the header. --- ## 2026-09-10 — Defaults and chrome * **All** is the default episode filter and the first tab; Unread, Downloaded and Flagged follow. Opening a feed and seeing nothing because everything in it was read is a poor first impression. * **OPML import/export moved into Settings**, under a Subscriptions heading, and the OPML button is gone from the sidebar -- it is a thing you do once in a while, not a daily control. * **The feed actions are pills with icons**, and Unsubscribe is pushed to the far end in a quiet style: it sat next to Settings looking exactly like it, one slip away from losing a feed. --- ## 2026-09-10 — Phone layout The UI was unusable on a phone, starting with the worst of it: the ☰ button lived inside the player bar, which is hidden until something plays, so there was no way to reach the feed list at all. It now sits in a small bar at the top of the main pane that is always there, and the sidebar slides over the page with a scrim to tap away. The rest: * **One pane at a time.** The three-pane split becomes a list, with the item text taking the whole screen over it and a `← Episodes` button back. The divider is hidden. * **No sideways scrolling.** A grid column is min-content wide by default, so one long headline ("davewiner/hackerNewsStars") dragged the entire page off the right edge -- `min-width:0` down the shell/main/wrap chain, `overflow-wrap:anywhere` on headings, and an explicit `minmax(0,1fr)` column for the OPML child list. * **The player stacks**: title row on top, transport and seek bar below, and it paints above the reading pane so it stays reachable. * Header artwork, buttons and log rows shrink to fit; tap targets go to 38px. A Playwright case at 390x844 locks in the three things that actually broke: the burger is visible with nothing playing, the page does not scroll sideways, and an item opens and closes over the list. --- ## 2026-09-10 — Marking a subscription read An OPML subscription's page now has **Mark all read**, sitting where every other feed keeps it -- before Unsubscribe. The fix is in the handler rather than the button: `read-all` resolves the feeds whose group is the given id (through `subscriptions()`, so a child promoted to config counts too) and marks those, since the subscription's own row holds no entries and marking it did nothing. --- ## 2026-09-10 — A folder counts what it holds An OPML subscription has no entries of its own, so its row always read `0 unread` no matter how much was waiting inside it. Unread and saved are now summed from the feeds it holds -- from all of them, not just the ones a search filter left showing, so the number doesn't move as you type. The badge caps at `999+` (the real figure is in its tooltip); a four-digit count ate the title beside it. --- ## 2026-09-10 — Sidebar alignment The feed list had four different left edges: a row with no children skipped the chevron entirely, so its artwork sat a chevron-width left of a folder's; children then used a different indent *and* a smaller icon. Nothing lined up with anything. Every row now reserves the chevron slot whether or not it opens (an empty one is `pointer-events:none`, so the click falls through to the row), all artwork is one size, and nesting reads from the indent alone. Both label lines are `display:block` on a shared line-height instead of an inline baseline, and the unread count has a `min-width` so three-digit feeds don't shove the title. Rows came out shorter, so more feeds fit without scrolling. --- ## 2026-09-10 — An item's picture An item's artwork now resolves in order of how deliberate the source is: `itunes:image`, then Media RSS `media:thumbnail`, then a `media:content` that declares itself an image, and finally an image **enclosure**. That last one matters here -- Substack puts each article's header picture in an ``, which is why those blog entries had no artwork despite carrying one all along. Audio enclosures are never mistaken for pictures. Backfilling needed the validators cleared first: `record_entry` fills a missing image on update, but a 304 skips parsing entirely, so the feeds would have kept their blank squares. (The self-heal added earlier only fires when a feed has *zero* entries, which was not the case here.) Result across the library: 325 of 3470 entries now carry their own picture, 13 feeds where every entry has one, 69 feeds that publish no per-item image at all -- those fall back to the feed's artwork, which is the intended behaviour rather than a gap. --- ## 2026-09-10 — Database cleanup, and the 304 trap it walked into Cleaned up on request: removed the `CT Log Archive Torrents` folder (123 preallocated files from the abandoned 1.6 TB tuscolo torrent, **19 GB real** on disk, no row referencing any of it, feed no longer subscribed), the unsubscribed ct-log feed's rows, and every OPML-derived entry/enclosure that held no file so a rescan could rebuild them with the current parser. Kept every row holding a file, so nothing on disk was orphaned, and kept the skipped/reaped history, without which the next scan would re-download the 149 images just deleted. Disk 22 GB -> 3.8 GB, database 22.8 MB -> 5.2 MB. Correction worth recording: mid-download I checked that torrent folder, saw `du -sh` report 8 KB, and told Ray it was sparse with nothing written. By the time it was abandoned it had allocated 19 GB. The reassurance had a shelf life I did not mention. **Then "Abort Retry Fail is empty".** The cleanup deleted entries but left each feed's ETag/Last-Modified. The rescan sent them, servers answered 304 (120 times in the log), the daemon skipped parsing, and 57 feeds stayed empty -- and would have until a publisher happened to change something. Cleared the validators on the empty feeds and rescanned: entries 1524 -> 3009, feeds with content 19 -> 64, Abort Retry Fail back to 20. Fixed in code so it cannot recur: a 304 arriving while the feed has **zero stored entries** means the validator has outlived the data -- a restore, a manual edit, a cleanup. The daemon now believes the database over the validator, drops it and asks again. Proved live by deleting a feed's entries, leaving its ETag, and rescanning: 20 entries rebuilt, self-heal logged once. **"not a wanted media type"** was internal jargon reaching the UI, and it was stored in `last_error` so an ordinary filter decision rendered in red as though something had failed. Reworded to "not audio or video", 144 existing rows updated, and the UI now paints a reason red only when the state is actually `error`. --- ## 2026-09-10 — Multiple enclosures per item, and viewing without downloading **View without downloading.** A non-audio/video enclosure now carries a View link opening in a new tab: the publisher's own URL when nothing is downloaded, the local copy at `/media/` when it is. Deliberately a direct link rather than a proxy — relaying arbitrary URLs through the daemon would make it a fetch-anything service. **Multiple enclosures.** Probed before assuming, and the result was worse than expected: the `rss` crate models an item as having at most one enclosure (which is what RSS 2.0 says) and when a feed carries several it silently keeps **the last**, dropping the rest. So a two-file item lost its first file entirely. `enclosures_by_item()` reads them straight from the XML with quick-xml, in document order, unescaping attribute values — a feed URL's `&` arrives as `&`, so skipping that would corrupt every query string. It falls back to the parsed enclosure if the scan and the parser disagree on item count. Atom already collected all `rel="enclosure"` links. The row now summarises the enclosure you would act on (playable, else downloaded, else first) and says "+N more files"; the pane below lists them all. **A real limitation surfaced by a broken fixture.** Two browser tests failed with zero enclosures in the detail pane. Not the new scanner — verified by running it against the fixture files directly, which was right every time. The cause: my fixtures pointed two feeds at the *same* enclosure URL, and `enclosures.url` is UNIQUE across the whole database, so whichever feed is scanned first claims it and the other's entry gets nothing. That is the dedupe working as designed, but it means **two feeds legitimately sharing a media URL will only ever show it under one of them** — worth knowing, and worth revisiting if a network feed and a show feed ever overlap. --- ## 2026-09-10 — A file is not the same as a playable file Reported: "Abort Retry Fail still shows downloaded and audio playback UI". The media-type filter stopped *new* image enclosures being fetched, but those 20 JPEGs were already on disk from before it existed — and the UI gave a play button and an `