Compare commits
48 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9aae3097e7 | |||
| 2ff2074755 | |||
| 1698cf8d1e | |||
| b94a74ef15 | |||
| 9a8a3c696f | |||
| bedf64e645 | |||
| 586d2c07a1 | |||
| 2ba83c3aed | |||
| 1352f0d54d | |||
| a958f7cb37 | |||
| 457a58dcc5 | |||
| 2af57065c6 | |||
| 564b011c7a | |||
| 9eb7aadced | |||
| c6bceaef37 | |||
| dc63d6acaf | |||
| 8937f35f00 | |||
| 0990f2a90d | |||
| 0cc002cdfa | |||
| 2c9e899762 | |||
| 2e416f96cf | |||
| 9269aa99f7 | |||
| ae8123250b | |||
| 2c34a144ba | |||
| 669e8b5124 | |||
| 5362436766 | |||
| 0efc49519c | |||
| 57dcba2d1a | |||
| e95cccc66f | |||
| 7295be8b25 | |||
| f1b0d97b81 | |||
| ae47e31a97 | |||
| 8a309eb652 | |||
| 1a2b0d87c6 | |||
| d7fac2d0e7 | |||
| df9b7645d6 | |||
| f3825cfc57 | |||
| c0f4b0bcb2 | |||
| 8b8b48302b | |||
| 5d3fdde4da | |||
| 8784d0a3fd | |||
| 5e95557cbb | |||
| 6114add4a6 | |||
| c47c224372 | |||
| 686851b448 | |||
| f0d03c79c8 | |||
| 7df4ee7dde | |||
| d46ec73261 |
305
CHANGELOG.md
Normal file
305
CHANGELOG.md
Normal file
@@ -0,0 +1,305 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
The long form, with what was wrong before and how it was found, is in
|
||||
[docs/history.md](docs/history.md).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.5.2] - 2026-09-12
|
||||
|
||||
### Added
|
||||
|
||||
- Settings → Users and `ipx user list` show when each account was added and when it last signed
|
||||
in, to the hour.
|
||||
- `ipx user rename <name> <new name>` renames an account and keeps its feeds, read state and admin
|
||||
rights. An account made before the proxy was set up can take the name the proxy signs it in as.
|
||||
|
||||
### Changed
|
||||
|
||||
- Directory and Popular list the feeds inside an OPML one by one, and no longer the OPML itself,
|
||||
so you can subscribe to just the shows you want.
|
||||
- The database no longer records when subscriptions and sign-in sessions were created. Nothing
|
||||
ever read it, and an existing database drops the columns on its next start.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Show notes that the podcast's host cut off in the middle of a tag no longer open with a scrap of
|
||||
HTML: the item's other copy of its notes is used instead, from the next time the feed changes.
|
||||
Daily Meditation Podcast had 57.
|
||||
- Docker no longer shows ipodderx as starting, or calls it unhealthy, while it scans or downloads:
|
||||
`ipx status` answers at once instead of waiting for the job in progress to finish.
|
||||
- Signing out after signing in through Cloudflare Access no longer lands on ipodderx's own password
|
||||
page. With the new `sign_out_url` set, Sign out ends the Access session, and the password page
|
||||
sends anyone the proxy signs in straight to their feeds.
|
||||
- The sign-in guide, `docs/sso.md`, describes the setup ipodderx.sdf1.net really runs: Authentik as
|
||||
Cloudflare Access's identity provider, and how to find the address ipx has to trust. It had never
|
||||
been checked against a real setup, and pointed at the wrong address.
|
||||
|
||||
## [0.5.1] - 2026-09-12
|
||||
|
||||
### Fixed
|
||||
|
||||
- The triangle that opens an OPML or Patreon folder was cramped against the folder's art. It has
|
||||
more room now, and a wider target to click.
|
||||
|
||||
## [0.5.0] - 2026-09-12
|
||||
|
||||
### Added
|
||||
|
||||
- A Patreon token pasted into Add feed, or a creator's RSS link without `&show=`, becomes a folder
|
||||
of that creator's shows, kept in step on every scan like a subscribed OPML. A creator with only
|
||||
one show stays a plain feed. One already added as a single long feed is split into its shows on
|
||||
its next scan, keeping its files and what you had read.
|
||||
- Add feed has an "Allow items marked explicit" box, so a new feed's first scan no longer skips
|
||||
every explicit item.
|
||||
|
||||
### Changed
|
||||
|
||||
- Unread counts, unread dots and download progress are amber, the colour of the icon's EQ bars.
|
||||
Blue is kept for the primary action and links, so a count no longer looks like a button.
|
||||
- What is playing is marked by small EQ bars, in its row and in the player. They move only while it
|
||||
plays.
|
||||
- A folder in the sidebar shows its first four shows' art as a mosaic, and its shows sit under its
|
||||
title. Only folders have a triangle, so every feed lines up with Directory and Popular above.
|
||||
- Feeds without art get initials in a colour of their own, instead of all the same grey.
|
||||
- The Flagged tab is called Kept, as the Keep button and Settings already said.
|
||||
- A feed's header is one short line; when it checks next is in its tooltip.
|
||||
- The item list takes more of the window, and the Files pane shows only when the item has files.
|
||||
- Column headings and tags are in sentence case, and fewer things are bold.
|
||||
- Unsubscribe is a round button beside the feed's other actions.
|
||||
- Export and Import in Settings say what they do.
|
||||
- The sign-in page shows the original icon large.
|
||||
- Nothing animates when your system asks for reduced motion.
|
||||
- The pages are about 90 KB smaller: the icon is served once instead of written into each.
|
||||
- A web token generated for a new install is 64 characters instead of 32.
|
||||
- The README is a short overview of what ipx does and how to run it, and points into `docs/` for
|
||||
the rest. It still described the layout from before 0.4.0.
|
||||
|
||||
### Removed
|
||||
|
||||
- The systemd units in `contrib/`. Run ipx with Docker, or point a unit of your own at
|
||||
`ipx daemon`.
|
||||
- Upgrading from before 0.3.0 directly: what was read, kept or part-played before accounts is no
|
||||
longer carried over to the admin, and OPML feeds that old versions wrote into `config.toml` are
|
||||
no longer moved out of it. Upgrade through 0.4.0 first.
|
||||
- `interval_mins` in `config.toml` is ignored; use `schedule`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- The feed list works from the keyboard: Tab reaches every feed, Enter opens it, and Right and Left
|
||||
open and close a folder. Every button shows where the focus is, including in the toolbar, which
|
||||
used to clip the ring.
|
||||
- "1 items" reads "1 item".
|
||||
- A selected feed without art no longer loses its initials tile in Dark and Light.
|
||||
- The player shows the feed's initials when there is no art, not the episode's.
|
||||
- Turning on Allow explicit, or changing keywords or auto-download, brings back what those settings
|
||||
had skipped on the feed's next scan. Before, an item was judged once, when first seen, and a
|
||||
skipped one stayed skipped whatever you changed.
|
||||
- Feeds inside an OPML or a Patreon creator follow your settings on the folder unless you set their
|
||||
own, as the folder's settings dialog said they did. Before, the folder's settings reached nothing
|
||||
inside it.
|
||||
- A new feed no longer takes the name of one you removed earlier and shows that feed's old items.
|
||||
Re-adding the same feed still gets its old name, and its history, back.
|
||||
|
||||
## [0.4.0] - 2026-09-11
|
||||
|
||||
### Added
|
||||
|
||||
- A Classic theme after the 2004 Mac app, beside Dark and Light: brushed-metal toolbar, Aqua
|
||||
blue selection, red unread badges, a striped table and Lucida Grande. The theme button steps
|
||||
through all three and remembers the choice.
|
||||
- A toolbar across the top, after the original iPodderX: add and unsubscribe, play, mark read
|
||||
and keep for the selected item, scan, a search box for what is showing, and Settings and Log.
|
||||
- Directory, Popular and All Subscriptions at the top of the feed list, opening in the main pane.
|
||||
Directory lists every feed anyone here subscribes to, A to Z (`GET /api/directory`). All
|
||||
Subscriptions lists every item from every feed you subscribe to (`GET /api/entries`).
|
||||
- Items show as a table (unread, kept, title, feed, file, size, published) with a Files pane beside
|
||||
it, and a status bar with the totals.
|
||||
- Mark everything read from All Subscriptions, across every feed you subscribe to
|
||||
(`POST /api/read-all`). It asks first. All Subscriptions can also check every feed from its header.
|
||||
- Click a column heading in the item table to sort by it (kept, title, feed, file type, size,
|
||||
published); click again to reverse. The server sorts, so it covers the whole list, not just the
|
||||
fifty shown, and the choice is remembered.
|
||||
|
||||
### Changed
|
||||
|
||||
- Popular shows the top 10, not 20, and counts everyone, you included. Your own feeds stay on it,
|
||||
marked Subscribed, and clicking one opens it.
|
||||
- Adding a feed scans it straight away, and an OPML import that added feeds scans them, so their
|
||||
items show without pressing Scan.
|
||||
- A file deleted to save space, or by hand, looks as if it was never downloaded: no "reaped"
|
||||
label, just the Download button. The retention summary says "deleted", not "reaped".
|
||||
- Buttons are icons, with the words in their tooltips: the Files pane (save, delete, view,
|
||||
download), an item's own buttons (mark read, keep, open the original), the feed header (scan,
|
||||
download latest, mark all read, settings, unsubscribe), and the Settings, feed settings and
|
||||
Download latest dialogs (save, download, cancel). The icons are Font Awesome Free, embedded as
|
||||
SVG: only the ones used, no font to download, and nothing fetched from anyone else. They
|
||||
replace font characters such as ⟳ ⤓ ↗, which came out thin and tiny and differed from font to
|
||||
font. Keep is a flag everywhere, as it was in the original, and mark unread is an envelope.
|
||||
- One meaning per icon. Minus unsubscribes, x closes or cancels, plus adds or subscribes, and a
|
||||
dialog's confirm button carries the icon of what it does. The feed header's unsubscribe was an x
|
||||
and read as closing the page. The remaining word buttons are icons too:
|
||||
- Log, Add feed, Users, Unsubscribe and OPML.
|
||||
- Popular's Subscribe, Copy and Sign out.
|
||||
- The Subscribed label in Popular, the Directory and Add feed, which is now a green check.
|
||||
- The player's back, play, forward and close, which were font characters, and the folder arrow.
|
||||
- The toolbar's read and keep buttons show the selected item's state, with the same icons as the
|
||||
item's own buttons. Play, read and keep sit together, and Scan sits with add and unsubscribe.
|
||||
- An OPML subscription's page has the same header as a feed's, with its buttons in the same places.
|
||||
- The item table's size has its own column, apart from the file's type, and shows KB for small
|
||||
files instead of "0 MB". The Item heading is now Title.
|
||||
- A file's type is an icon (audio, video, image, PDF, torrent, other), green once it is
|
||||
downloaded and red when the download failed, with the details in its tooltip. One icon per
|
||||
row keeps the column lined up. The DOWNLOADED and PENDING labels are gone.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Playing a file from the Files pane played it twice at once, in the pane and in the player bar.
|
||||
The pane has a play button now, and the player bar is the only player.
|
||||
- The password box in Manage users was white in the dark theme.
|
||||
- An item with no date showed a stray dot in its details.
|
||||
- Escape did not close a dialog while the cursor was in one of its boxes, so Add feed, which opens
|
||||
in its URL box, could not be closed with Escape.
|
||||
|
||||
### Security
|
||||
|
||||
- Feeds from paid-feed services (Patreon, Supercast, Supporting Cast, Glow, Memberful) are never
|
||||
listed in Popular or the Directory. A Supercast feed, which keeps its key in the URL's path
|
||||
rather than the query, was being listed.
|
||||
|
||||
## [0.3.0] - 2026-09-11
|
||||
|
||||
### Added
|
||||
|
||||
- Add feed lists what other people on this server subscribe to, most subscribers first, and
|
||||
subscribes you by id (`GET /api/popular`, `POST /api/popular/{id}`). Feeds from an OPML, and
|
||||
feeds with a login or a key in their URL, are never listed.
|
||||
- Upload an OPML file to import, beside the paste box. The page checks it looks like OPML before
|
||||
sending it and clears the picker afterwards.
|
||||
- Settings → Manage users: add and remove accounts, and choose who is an admin
|
||||
(`GET`/`POST /api/users`, `PATCH`/`DELETE /api/users/{id}`).
|
||||
- Per-user subscriptions, and per-user read, starred and playback state. The existing library is
|
||||
adopted by the admin on first start.
|
||||
- Subscribing to a feed someone else already has costs no second fetch and no second copy. Scanning
|
||||
merges every subscriber's wants.
|
||||
- Delete on a shared feed reads **Delete for everyone**, and the server answers `409` while anyone
|
||||
else has starred the item or not played it (`?force=true` overrides).
|
||||
- A shared feed's header says how many other people read it.
|
||||
- `docs/` for configuration, the CLI, users, SSO and architecture, and `CLAUDE.md` for anyone
|
||||
working on the code.
|
||||
- Browser tests for OPML import and export by every route, user admin, unread ordering, and
|
||||
`ipx import`/`ipx export`.
|
||||
|
||||
### Changed
|
||||
|
||||
- Feeds inside an OPML subscription list the ones with unread items first.
|
||||
- OPML import subscribes you to every feed in the file. `ipx import` subscribes the first admin.
|
||||
- OPML export lists only your own subscriptions.
|
||||
- Production runs as a Docker image pushed to `192.168.1.130:5000` and recreated with
|
||||
`docker compose`.
|
||||
- This changelog follows Keep a Changelog. The long-form entries moved to `docs/history.md`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Importing another account's OPML export subscribed nobody and reported "Imported 0 feed(s)".
|
||||
Feeds it added had no subscriber, so they were never scanned.
|
||||
- Importing something that is not OPML answered `500`. It is now `400` "that is not an OPML file",
|
||||
refused before anything changes.
|
||||
- Starring stopped protecting a file from the quota and age sweeps once read state became per-user.
|
||||
|
||||
### Security
|
||||
|
||||
- The log is admin-only (`GET /api/logs` answers `403`, and the Log button is hidden). It names
|
||||
every account, every feed and every failed sign-in.
|
||||
- OPML export no longer hands anyone signed in the whole catalogue, including other people's
|
||||
private feed URLs.
|
||||
|
||||
## [0.2.0] - 2026-09-10
|
||||
|
||||
### Added
|
||||
|
||||
- Web UI served by the daemon: plain HTML and JS compiled into the binary, with feeds, items,
|
||||
filters, search, sanitised show notes and live progress over SSE.
|
||||
- Player bar with resume, speed, keyboard shortcuts and lock-screen controls.
|
||||
- Three-pane layout: feeds beside, items above, and the selected item's text and files below.
|
||||
- Phone layout.
|
||||
- Accounts and sign-in: Argon2id passwords, session cookies, `ipx user add|list|passwd|rm`, and a
|
||||
trusted proxy header for Cloudflare Zero Trust or Authentik (`docs/sso.md`).
|
||||
- Subscribing to an OPML: it is re-read every scan and its feeds show as a folder. A feed dropped
|
||||
from it is removed unless it has downloads.
|
||||
- Scheduling: a global interval with per-feed overrides (`every 30m`, `4h`, `1d`, `2w`).
|
||||
- `[general] media_types`, default audio and video, and `max_new_per_check`, default 3.
|
||||
- Every enclosure of an item, a View link for files that are not audio or video, and per-item
|
||||
artwork.
|
||||
- In-app log view with Daemon I/O, Scans and HTTP tabs.
|
||||
- Mark all read on an OPML subscription.
|
||||
- Editable feed URL with a copy button.
|
||||
- Docker image whose healthcheck goes through the control socket.
|
||||
- The iPodderX name, icon, and a colour scheme taken from the icon.
|
||||
- `tests/page-smoke.js` and a Playwright browser suite.
|
||||
|
||||
### Changed
|
||||
|
||||
- Global settings and scan schedules are admin-only. The per-feed schedule picker is gone.
|
||||
- Feeds from an OPML live in the database, not `config.toml`.
|
||||
- Opening an item marks it read. Playing it marks it read only at the end or past 90%.
|
||||
- "Episodes" became "items", since half the library is text.
|
||||
- A burst of scan events causes one refresh, not one per feed.
|
||||
- All is the default filter. OPML import and export, Settings and Log moved out of the header.
|
||||
- Torrents run detached, two at a time.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Download fetched the next queued episodes instead of the one clicked.
|
||||
- Pressing play made an item vanish from the Unread list.
|
||||
- Clearing a folder, schedule or cap from the UI did nothing.
|
||||
- The daemon ignored SIGTERM until the current download finished.
|
||||
- A missing function stopped the page script and left the whole UI dead.
|
||||
- One download painted progress on every pending row.
|
||||
- A torrent could freeze scanning for up to an hour.
|
||||
- Downloading from an OPML feed failed with "belongs to unsubscribed feed".
|
||||
- Image enclosures were downloaded, counted as episodes and given a play button.
|
||||
- A feed whose entries had been deleted stayed empty, because the server kept answering `304`.
|
||||
- An item with several enclosures kept only the last.
|
||||
- OPML folders rendered open by default.
|
||||
- Mark read in the text pane recursed until the stack overflowed.
|
||||
- The Unread, Downloaded and Flagged filters answered `500` without a search term.
|
||||
- An OPML subscription always showed 0 unread.
|
||||
- Folder names kept doubled spaces where separators were stripped.
|
||||
- Sidebar rows had four different left edges.
|
||||
|
||||
### Security
|
||||
|
||||
- The web UI needs a token or a sign-in. The token is compared in constant time, and an empty
|
||||
token refuses to serve.
|
||||
- Show notes are sanitised with `ammonia`.
|
||||
- A proxy's user header is honoured only from an address in `trusted_proxies`.
|
||||
- A feed URL must be http(s), so `file:///etc/passwd` is refused.
|
||||
- Download folders are sanitised per path segment, so `../../etc/Show` cannot climb out.
|
||||
|
||||
## [0.1.0] - 2026-09-09
|
||||
|
||||
### Added
|
||||
|
||||
- `ipx`, a Rust rewrite of the iPodderX engine: TOML config, SQLite state, and `ipx list`, `add`,
|
||||
`rm` and `fetch`.
|
||||
- RSS and Atom parsing with conditional GET, `<ttl>` and basic auth.
|
||||
- Streaming downloads with explicit, keyword and per-scan filters, deduplicated by enclosure URL.
|
||||
- Quota and age retention that never touches a starred file, and `ipx reap [--dry-run]`.
|
||||
- `ipx daemon` with a JSON-lines Unix socket. CLI commands proxy to a running daemon.
|
||||
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
|
||||
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
|
||||
|
||||
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.2...main
|
||||
[0.5.2]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.1...v0.5.2
|
||||
[0.5.1]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.0...v0.5.1
|
||||
[0.5.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...v0.5.0
|
||||
[0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0
|
||||
[0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0
|
||||
[0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0
|
||||
[0.1.0]: https://git.sdf1.net/rays/ipodderx-rs/releases/tag/v0.1.0
|
||||
153
CLAUDE.md
Normal file
153
CLAUDE.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# Working on ipodderx-rs
|
||||
|
||||
Notes for whoever picks this up next. Read [docs/architecture.md](docs/architecture.md) for how the
|
||||
thing is built; this file is about working on it without repeating mistakes that have already been
|
||||
made here.
|
||||
|
||||
## Where things are
|
||||
|
||||
Production is the `iPodderX` container on Tower (192.168.1.130), the `ipodderx` service of the
|
||||
Arcane project `content`: `/mnt/fast/arcane/projects/content/compose.yaml`. That file is what runs;
|
||||
`docker-compose.yml` in this repo is a copy, and editing it changes nothing in production.
|
||||
|
||||
| | Host | In the container |
|
||||
|---|---|---|
|
||||
| Image | `192.168.1.130:5000/ipodderx:latest` | |
|
||||
| Config | `/mnt/fast/appdata/ipodderx/config.toml` | `/config/config.toml` |
|
||||
| Database | `/mnt/user/ipodderx/state.db` | `/data/state.db` |
|
||||
| Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` |
|
||||
| Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` |
|
||||
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | trusts `Cf-Access-Authenticated-User-Email` from `192.168.16.1`, the `content_default` gateway |
|
||||
|
||||
Deploying a change is: build and push the image, then pull it and recreate the container.
|
||||
|
||||
```sh
|
||||
docker buildx build --tag 192.168.1.130:5000/ipodderx:latest . --push
|
||||
docker compose -f /mnt/fast/arcane/projects/content/compose.yaml pull ipodderx
|
||||
docker compose -f /mnt/fast/arcane/projects/content/compose.yaml up -d ipodderx
|
||||
docker logs --tail 20 iPodderX
|
||||
```
|
||||
|
||||
**Name the service.** A bare `up -d` recreates every container in `content`, beets and immich
|
||||
included. Run `pull` before `up`, because `up` reuses whatever `latest` the host already has.
|
||||
|
||||
**A build that fails with `429 Too Many Requests` on a base image** is Docker Hub rate-limiting
|
||||
this host. There is no Docker Hub login here, and the build asks about `debian:bookworm-slim` and
|
||||
`rust:1-slim-bookworm` every time unless they are already stored locally. Pull them from Google's
|
||||
mirror and tag them; the build then uses the local copies without asking Docker Hub:
|
||||
|
||||
```sh
|
||||
docker pull mirror.gcr.io/library/debian:bookworm-slim
|
||||
docker tag mirror.gcr.io/library/debian:bookworm-slim debian:bookworm-slim
|
||||
docker pull mirror.gcr.io/library/rust:1-slim-bookworm
|
||||
docker tag mirror.gcr.io/library/rust:1-slim-bookworm rust:1-slim-bookworm
|
||||
```
|
||||
|
||||
Run those again now and then, or the local copies go stale.
|
||||
|
||||
The healthcheck runs `ipx status` against the control socket, so `(healthy)` in `docker ps` means
|
||||
the daemon answers there and can read its database, not just that the web port is up. The socket
|
||||
answers `status` itself instead of queuing it behind the worker's current job, so a long scan or
|
||||
download does not fail the check; it also means a worker stuck on one job would still pass. The
|
||||
container restarts on its own after a reboot.
|
||||
|
||||
Before the container, ipx ran by hand in code-server, with its files in `/config/.config/ipx/` and
|
||||
`/config/.local/share/ipx/`. Those are still there and the container does not read them. If you run
|
||||
a daemon by hand for testing, stop it with **`pkill -x ipx`, never `pkill -f ipx`**. `-f` matches
|
||||
the shell running the command and kills the session (exit 144). This has happened more than once.
|
||||
|
||||
## Before you touch the page
|
||||
|
||||
`web/index.html` is `include_str!`d into the binary, so **every page change needs a rebuild** before
|
||||
it is visible. It is one file: markup, CSS and script.
|
||||
|
||||
After any edit to it:
|
||||
|
||||
```sh
|
||||
node tests/page-smoke.js
|
||||
```
|
||||
|
||||
That loads the script against a stub DOM and checks every selector it wires at load actually
|
||||
exists. It exists because a patch once anchored on a deleted function, `String.replace` silently
|
||||
matched nothing, and the whole UI died with a `ReferenceError` while every server-side test passed.
|
||||
|
||||
Patching that file by guessing an anchor string has failed repeatedly. Read the exact block first
|
||||
(`sed -n 'START,ENDp'`), match it verbatim, and assert the replacement happened rather than hoping.
|
||||
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
cargo test # ~51 tests: parsing, filters, retention, schedules, SQL, per-user state
|
||||
node tests/page-smoke.js
|
||||
npx playwright test # 16 browser tests against a real daemon on fixture feeds
|
||||
```
|
||||
|
||||
Things about the browser suite that have cost time:
|
||||
|
||||
* It starts **its own daemon and database** under `/tmp/ipx-ui-test`, wiped once per run. Playwright
|
||||
re-imports the config in every worker, so `prepare()` guards on `TEST_WORKER_INDEX` — without
|
||||
that guard a worker deleted the database out from under the running daemon, which then kept
|
||||
serving from the unlinked inode while everything else saw an empty file.
|
||||
* Tests **share that daemon and run in order**. A test that opens an item marks it read and changes
|
||||
what later tests see. Write assertions that do not depend on what ran before, or normalise the
|
||||
state first.
|
||||
* Fixture feeds must not share an enclosure URL, because `enclosures.url` is globally unique and
|
||||
whichever feed is scanned first claims it.
|
||||
* `webServer` starts **before** `globalSetup`, which is why the fixture config is written at
|
||||
config-load time instead.
|
||||
|
||||
Non-trivial logic leaves one runnable check behind. Pure functions (`merge_policy`, `pick`,
|
||||
`matches_keywords`, `parse_interval`) are the easiest place to put it.
|
||||
|
||||
## Things that are true and easy to get wrong
|
||||
|
||||
* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
|
||||
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
|
||||
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
|
||||
and `position` columns from before accounts; two bugs came from queries still reading them
|
||||
(retention, and the entry pruner), and `migrate()` now drops them.
|
||||
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
|
||||
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
|
||||
derived and never written to config.
|
||||
* **One fetch serves everyone**, so scan policy is a union of subscribers' wants (`merge_policy`).
|
||||
Anyone wanting an item is enough to fetch it.
|
||||
* **The UI hiding a control is not enforcement.** Admin-only actions check `user.is_admin` in the
|
||||
handler and return `403`.
|
||||
* **A `tokio::select!` only races its branches at the point of selection.** A long download has to
|
||||
watch the shutdown channel itself; the daemon ignored SIGTERM for exactly this reason.
|
||||
* Only one daemon per socket. Removing the socket file defeats the guard and you get two daemons
|
||||
fighting over the database, with the stale one still holding the port.
|
||||
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
|
||||
different task. `ipx status` checks the control socket and the database; to see the worker
|
||||
getting through its jobs, watch for `scan complete` in the log.
|
||||
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's
|
||||
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on
|
||||
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait
|
||||
for `daemon started` in the log. Copy `state.db` aside before deploying one.
|
||||
|
||||
## House style
|
||||
|
||||
Comments explain **why**, not what. If a line looks odd, the comment says what went wrong without
|
||||
it. No emoji, no exclamation marks, no "obviously". Prose in the UI and docs is plain English and
|
||||
addressed to the person using it.
|
||||
|
||||
Every change gets one line under `## [Unreleased]` in [CHANGELOG.md](CHANGELOG.md), in its
|
||||
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) group: Added, Changed, Deprecated,
|
||||
Removed, Fixed or Security. Say it the way someone using ipx would notice it. When there is more to
|
||||
say, such as what was wrong before or what it cost to find out, write it up at the top of
|
||||
[docs/history.md](docs/history.md), dated. That record has been more useful than the git log more
|
||||
than once.
|
||||
|
||||
Cutting a release: rename `[Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD` and open a new empty
|
||||
`[Unreleased]` above it, bump `version` in `Cargo.toml`, tag the commit `vX.Y.Z`, and update the
|
||||
compare links at the bottom of the changelog.
|
||||
|
||||
Deliberate simplifications get a `ponytail:` comment naming the ceiling and the upgrade path, e.g.
|
||||
`// ponytail: global connection mutex, move to a pool if feed count makes it contend`.
|
||||
|
||||
## Known gaps
|
||||
|
||||
* Cloudflare's `Cf-Access-Jwt-Assertion` is not verified — ipx trusts the hop plus `trusted_proxies`
|
||||
(documented in [docs/sso.md](docs/sso.md)).
|
||||
* A feed's `<description>` subtitle is dropped whenever `content:encoded` exists, which loses
|
||||
Substack-style subtitles.
|
||||
34
Cargo.lock
generated
34
Cargo.lock
generated
@@ -436,17 +436,6 @@ dependencies = [
|
||||
"shlex",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfb"
|
||||
version = "0.14.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a347dcabdae9c31b0825fd6a8bed285ec9c2acb89c47827126d52fa4f59cece3"
|
||||
dependencies = [
|
||||
"fnv",
|
||||
"uuid",
|
||||
"web-time",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfg-if"
|
||||
version = "1.0.4"
|
||||
@@ -867,15 +856,6 @@ dependencies = [
|
||||
"dirs-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dirs"
|
||||
version = "7.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8d57d423b3c82e89b9a24ca3091fee61f456a26edbd28d26c65906f4bc1dcd8f"
|
||||
dependencies = [
|
||||
"dirs-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dirs-sys"
|
||||
version = "0.5.0"
|
||||
@@ -1608,15 +1588,6 @@ dependencies = [
|
||||
"serde_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "infer"
|
||||
version = "0.22.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f4200d433cbd5178df7797c9c2e75b348b728e39631cf14520d1e2fc424201f4"
|
||||
dependencies = [
|
||||
"cfb",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "intervaltree"
|
||||
version = "0.2.7"
|
||||
@@ -1634,7 +1605,7 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
|
||||
|
||||
[[package]]
|
||||
name = "ipx"
|
||||
version = "0.1.0"
|
||||
version = "0.5.2"
|
||||
dependencies = [
|
||||
"ammonia",
|
||||
"anyhow",
|
||||
@@ -1643,9 +1614,7 @@ dependencies = [
|
||||
"axum",
|
||||
"chrono",
|
||||
"clap",
|
||||
"dirs",
|
||||
"futures-util",
|
||||
"infer",
|
||||
"librqbit",
|
||||
"opml",
|
||||
"percent-encoding",
|
||||
@@ -1656,7 +1625,6 @@ dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
"tokio-stream",
|
||||
"toml",
|
||||
"tower",
|
||||
"tower-http 0.7.1",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "ipx"
|
||||
version = "0.1.0"
|
||||
version = "0.5.2"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
@@ -11,9 +11,7 @@ atom_syndication = "0.12.10"
|
||||
axum = "0.8.9"
|
||||
chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] }
|
||||
clap = { version = "4.6.6", features = ["derive"] }
|
||||
dirs = "7.0.0"
|
||||
futures-util = { version = "0.3.34", default-features = false, features = ["std"] }
|
||||
infer = "0.22.0"
|
||||
librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] }
|
||||
opml = "1.1.6"
|
||||
percent-encoding = "2.3.2"
|
||||
@@ -24,7 +22,6 @@ rusqlite = { version = "0.40.2", features = ["bundled"] }
|
||||
serde = { version = "1.0.229", features = ["derive"] }
|
||||
serde_json = "1.0.151"
|
||||
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
|
||||
tokio-stream = { version = "0.1.19", features = ["sync"] }
|
||||
toml = "1.1.5"
|
||||
tower = { version = "0.5.3", features = ["util"] }
|
||||
tower-http = { version = "0.7.1", features = ["fs"] }
|
||||
|
||||
259
README.md
259
README.md
@@ -1,238 +1,83 @@
|
||||
# ipodderx-rs
|
||||
|
||||
A headless podcatcher: scans RSS/Atom feeds, downloads enclosures (HTTP and BitTorrent),
|
||||
files them into per-feed folders, and reaps old episodes to stay under a disk quota.
|
||||
Runs as a one-shot CLI or as a daemon with a Unix-socket JSON event stream for a UI to attach to.
|
||||
A self-hosted podcatcher for a household. It checks your feeds, downloads the episodes, and serves
|
||||
a web UI modelled on the 2004 Mac app **iPodderX**, for any number of people sharing one copy of
|
||||
the files. One Rust binary, `ipx`, is both the daemon and the command line.
|
||||
|
||||
## Lineage
|
||||
It is a rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python engine
|
||||
behind iPodderX (2004-2008, Ray Slakinski & August Trometer).
|
||||
|
||||
This is a modern Rust rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the
|
||||
Python 2 engine behind **iPodderX** (2004-2008, Ray Slakinski & August Trometer), which was
|
||||
open-sourced under the MIT License in 2010.
|
||||
## What it does
|
||||
|
||||
What carries over: the feed scan and TTL handling, GUID/URL dedupe, per-feed and per-date download
|
||||
folders, keyword filters, the explicit-content filter, torrent enclosures, and "SmartSpace" -- the
|
||||
oldest-first disk quota reaper.
|
||||
- **The web UI.** It has a toolbar, and a feed list that opens with Directory, Popular and All
|
||||
Subscriptions. Items sit in a sortable table with a Files pane, and there is a player bar. It
|
||||
comes in Dark, Light and Classic themes, and works on a phone.
|
||||
- **Several people, one copy.** Each person has their own subscriptions and their own read, kept
|
||||
and playback state. There is one file on disk per episode, however many people want it. People
|
||||
sign in with a password or through a proxy (Cloudflare Zero Trust or Authentik), and admins
|
||||
manage accounts and settings.
|
||||
- **Scanning.** Feeds are checked on a schedule, globally or per feed, and a feed's own TTL is
|
||||
honoured. Keyword, explicit-content and media-type filters decide what is downloaded, with a cap
|
||||
on new downloads per scan.
|
||||
- **Downloads.** Files come over HTTP or BitTorrent and are filed into a folder per feed.
|
||||
Retention deletes the oldest files to stay under a disk quota or an age limit, and never touches
|
||||
an item someone has kept.
|
||||
- **OPML.** You can import and export your own subscriptions. You can also subscribe to an OPML
|
||||
URL, which keeps a whole list in step as a folder.
|
||||
|
||||
What does not: iTunes and iPhoto export via AppleScript, text-to-speech enclosures, the Windows
|
||||
WMP/COM paths, XML plists and Python pickles for state, the `directory.iPodderX.com` survey ping,
|
||||
3DES-encrypted preferences, and the `printMSG` stdout protocol (replaced by a JSON-lines socket).
|
||||
## Run it
|
||||
|
||||
## Quickstart
|
||||
With Docker:
|
||||
|
||||
```sh
|
||||
cargo install --path .
|
||||
|
||||
ipx add https://atp.fm/rss # names the feed from its own title
|
||||
ipx list
|
||||
ipx fetch # scan now
|
||||
ipx daemon # or run continuously, honouring each feed's <ttl>
|
||||
docker build -t ipodderx .
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Config lives at `~/.config/ipx/config.toml` (mode 0600, since it may hold feed passwords);
|
||||
state at `~/.local/share/ipx/state.db`. Override with `IPX_CONFIG` and `IPX_DATA_DIR`.
|
||||
Set `IPX_LOG=ipx=debug` for verbose logging on stderr.
|
||||
`docker-compose.yml` is set up for the author's own server. Point its `image` and its three volumes
|
||||
(`/config`, `/data` and `/downloads`) at yours first. The UI is on port 8099. BitTorrent uses 6881
|
||||
over TCP and UDP. Files are written as `PUID`/`PGID`, 99:100 by default.
|
||||
|
||||
## Commands
|
||||
|
||||
| command | what it does |
|
||||
|---|---|
|
||||
| `ipx add <url> [--folder X] [--keywords a,b]` | subscribe; the id comes from the feed title |
|
||||
| `ipx rm <feed>` | unsubscribe; downloads and history are kept |
|
||||
| `ipx list` / `ipx status` | subscriptions and their state |
|
||||
| `ipx fetch [FEED] [--force]` | scan; `--force` ignores the TTL |
|
||||
| `ipx reap [--dry-run]` | run retention now |
|
||||
| `ipx import/export <file.opml>` | move subscriptions in or out |
|
||||
| `ipx daemon` | scheduler plus the control socket |
|
||||
|
||||
Any command with a wire form probes the socket first: if a daemon is running it does the work,
|
||||
and the CLI just renders the events it streams back. `--local` forces in-process execution.
|
||||
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[general]
|
||||
download_dir = "~/Podcasts"
|
||||
socket = "/run/user/1000/ipx.sock" # default: $XDG_RUNTIME_DIR/ipx.sock
|
||||
interval_mins = 60 # default poll; a feed's own <ttl> wins when longer
|
||||
organize = "feed" # "feed" | "date"
|
||||
max_total_gb = 50 # 0 = unlimited
|
||||
max_age_days = 30 # 0 = keep forever
|
||||
max_new_per_check = 3 # per feed, per scan. 0 = unlimited (pulls whole back catalogues)
|
||||
media_types = ["audio", "video"] # what downloads automatically. Anything else is still
|
||||
# listed and can be fetched by hand -- blog feeds put article
|
||||
# images in enclosures. Empty takes everything.
|
||||
|
||||
[torrent]
|
||||
enabled = true
|
||||
seed_ratio = 1.0 # stop seeding at this ratio ...
|
||||
seed_time_mins = 60 # ... or after this long, whichever comes first
|
||||
port_range = "6881-6889"
|
||||
stall_mins = 30 # give up on a torrent making no progress
|
||||
|
||||
[feeds.atp]
|
||||
url = "https://atp.fm/rss"
|
||||
folder = "Accidental Tech Podcast" # default: the feed title
|
||||
keywords = ["deep dive"] # OR across keywords, AND within one
|
||||
allow_explicit = false
|
||||
auto_download = true
|
||||
max_new_per_check = 3 # the rest wait for the next scan
|
||||
username = "ray" # optional HTTP basic auth
|
||||
password_env = "IPX_ATP_PASS" # or a literal `password`
|
||||
```
|
||||
|
||||
Retention keeps files that are `flagged` in the database, and deletes read episodes before unread
|
||||
ones, oldest first.
|
||||
|
||||
## Socket protocol
|
||||
|
||||
Newline-delimited JSON over a Unix socket, both directions.
|
||||
From source:
|
||||
|
||||
```sh
|
||||
$ printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
|
||||
{"ev":"feed_start","feed":"atp"}
|
||||
{"ev":"progress","feed":"atp","url":"...","file":"ep1.mp3","done":8192,"total":3000000}
|
||||
{"ev":"download_done","feed":"atp","url":"...","path":"...","bytes":3000000}
|
||||
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
|
||||
{"ev":"scan_done","feeds":1}
|
||||
cargo build --release
|
||||
./target/release/ipx daemon
|
||||
```
|
||||
|
||||
Commands: `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `status`.
|
||||
Events: `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
|
||||
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
|
||||
`scan_done`, `reap_done` and `status` are terminal -- a client that asked for work stops there.
|
||||
The first start creates **admin / ipodderx**. Sign in at `/login`, then change it:
|
||||
|
||||
Progress is throttled to whole percents. The stream is a broadcast, so a client attached to a busy
|
||||
daemon also sees that daemon's other work.
|
||||
```sh
|
||||
echo -n 'a good password' | ipx user passwd admin
|
||||
```
|
||||
|
||||
## OPML
|
||||
The UI is plain HTTP, so put TLS in front of it if it is reachable from outside your network.
|
||||
|
||||
Two different things, both supported:
|
||||
## Documentation
|
||||
|
||||
**Importing and exporting** a file copies subscriptions in or out once — `ipx import subs.opml`,
|
||||
`ipx export subs.opml`, or the OPML button in the UI.
|
||||
|
||||
**Subscribing to an OPML URL** is a live subscription, as iPodderX had. Add the OPML's URL like any
|
||||
other feed; every scan re-reads it and keeps your feed list in step. An OPML is recognised by its
|
||||
content, so a URL without a `.opml` extension still works.
|
||||
|
||||
The feeds inside it are **not written to `config.toml`** — the OPML is the source of truth, so they
|
||||
are re-derived each scan and held in the database. Your config keeps only what you chose. They show
|
||||
as a collapsible folder in the sidebar, download into one nested folder, and newly listed ones are
|
||||
scanned in the same run rather than waiting for the next interval. They inherit the subscription's
|
||||
settings; change anything on one and it gets its own config entry from then on.
|
||||
|
||||
When a feed drops out of the OPML upstream:
|
||||
|
||||
| it has downloads | what happens |
|
||||
| | |
|
||||
|---|---|
|
||||
| no | unsubscribed and removed from the config |
|
||||
| yes | kept, flagged in the UI as no longer listed |
|
||||
|
||||
A downloaded file is never left behind with nothing explaining where it came from.
|
||||
|
||||
## Web UI
|
||||
|
||||
```toml
|
||||
[web]
|
||||
enabled = true
|
||||
bind = "0.0.0.0:8080" # 127.0.0.1:8080 by default
|
||||
token = "" # generated and written back on first run
|
||||
```
|
||||
|
||||
`ipx daemon` then serves it in the same process (`ipx daemon --web ADDR` overrides the bind for one
|
||||
run). On first start it mints a token, saves it to config.toml, and prints the URL to open:
|
||||
|
||||
```
|
||||
web ui token generated. Open:
|
||||
http://0.0.0.0:8080/?token=1f4c…
|
||||
```
|
||||
|
||||
`?token=` sets a year-long cookie, so you only paste it once per browser. Everything is behind that
|
||||
token, including `/media/...` — a cookie rather than a header precisely because an `<audio>` element
|
||||
cannot send headers.
|
||||
|
||||
Browse feeds, read show notes, play episodes in the browser (Range requests are served, so seeking
|
||||
works), download or delete individual files, mark episodes read or flag them to keep, and edit a
|
||||
feed's folder/keywords/explicit/auto-download/limit settings. Config edits are written to
|
||||
config.toml and hot-reloaded — no daemon restart.
|
||||
|
||||
Show notes are feed-supplied HTML from an untrusted source; they are sanitized with `ammonia`
|
||||
server-side before they reach the page.
|
||||
|
||||
**It is plain HTTP.** On a LAN bind, the token and everything else crosses the network in the
|
||||
clear — and a feed URL can itself contain a credential (Patreon's, for one, carries an auth token).
|
||||
Put it behind a reverse proxy with TLS if that matters to you.
|
||||
|
||||
## Layout
|
||||
|
||||
Feeds down the side; the selected feed's episodes across the top; the selected episode's text and
|
||||
its enclosures in the pane below, which is where you play, download or delete them. The divider
|
||||
between the two panes drags and the position is remembered. An OPML subscription's page instead
|
||||
lists the feeds inside it.
|
||||
|
||||
An item may carry several enclosures. All of them appear in the pane below; the row summarises the
|
||||
one you would act on and notes how many others there are. Anything that is not audio or video gets a
|
||||
View link — opening the publisher's copy, or the local one once downloaded — rather than a player.
|
||||
|
||||
## Log view
|
||||
|
||||
The **Log** button in the sidebar shows the running daemon's output live, in four tabs:
|
||||
**Daemon I/O** is the control protocol itself — every command in and every event out, as JSON;
|
||||
**Scans** is feed and download activity; **HTTP** is web requests; **All** is everything: feed scans, downloads,
|
||||
torrent activity and every HTTP request, with level and text filters and a copy button. It reads a
|
||||
2000-line ring buffer held inside the process (`/api/logs`), not a file — so it works the same under
|
||||
Docker, where logs go to stdout and there is no file to tail. The buffer keeps `debug` even when the terminal does not, so protocol traffic and routine
|
||||
skips are there without making stderr unreadable — `IPX_UI_LOG` changes what it captures and
|
||||
`IPX_LOG` what reaches stderr.
|
||||
| [docs/configuration.md](docs/configuration.md) | Every config key, path and environment variable |
|
||||
| [docs/cli.md](docs/cli.md) | Every command, including `ipx user` |
|
||||
| [docs/users.md](docs/users.md) | Accounts, and what several people share |
|
||||
| [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik |
|
||||
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API |
|
||||
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
|
||||
| [docs/history.md](docs/history.md) | How it was built, with what was wrong and why |
|
||||
| [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed |
|
||||
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
cargo test # the server: parsing, filters, retention, schedules, SQL
|
||||
cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state
|
||||
node tests/page-smoke.js # the page script loads without throwing
|
||||
npx playwright test # a real browser against a real daemon
|
||||
npx playwright test # a real browser against a real daemon on fixture feeds
|
||||
```
|
||||
|
||||
The Rust tests cover the server and the smoke test catches a script that fails to load, but neither
|
||||
can see a wrong selector, a handler that runs and does nothing, or a page that renders empty — which
|
||||
is what has actually slipped through. The Playwright suite drives a headless browser against a
|
||||
daemon started on fixture feeds, and each test maps to a bug that reached a user: the page rendering
|
||||
empty, a dead theme toggle, settings not persisting, episode metadata, filter tabs, the feed URL
|
||||
field and its copy button, the log tabs, and OPML folders.
|
||||
|
||||
`npm install` gets the test runner; the browser itself comes from
|
||||
`npx playwright install --with-deps chromium` (in `install.sh`).
|
||||
|
||||
## Docker
|
||||
|
||||
```sh
|
||||
docker compose up -d # builds the image and starts it
|
||||
docker compose logs -f ipx # the token is printed on first start
|
||||
```
|
||||
|
||||
`docker-compose.yml` mounts `./config`, `./data` and a downloads directory, publishes 8099 for the
|
||||
UI and 6881 (TCP **and** UDP — DHT needs the UDP side), and sets `PUID`/`PGID` to `99:100` so files
|
||||
land owned the way Unraid shares expect. On first start the entrypoint writes a config bound to
|
||||
`0.0.0.0`, since a container's loopback is not reachable from outside it, and prints the URL with
|
||||
its generated token.
|
||||
|
||||
The healthcheck runs `ipx status`, which goes through the control socket to the command worker — so
|
||||
it catches a daemon that is alive but wedged, not merely one that has died.
|
||||
|
||||
## Running it as a service
|
||||
|
||||
`contrib/` has a systemd user unit for the daemon, and a timer plus one-shot service if you would
|
||||
rather run periodic scans with no daemon (in which case there is no socket for a UI to attach to).
|
||||
`npm install` gets the test runner, and `npx playwright install --with-deps chromium` gets the
|
||||
browser.
|
||||
|
||||
## License
|
||||
|
||||
MIT. See [LICENSE](LICENSE).
|
||||
|
||||
## Signing in
|
||||
|
||||
A new database starts with **admin / ipodderx** at `/login`; change it with
|
||||
`echo -n <password> | ipx user passwd admin`. Accounts are managed with `ipx user add|list|passwd|rm`.
|
||||
|
||||
To let Cloudflare Zero Trust or Authentik do the authenticating instead, see
|
||||
[docs/sso.md](docs/sso.md).
|
||||
MIT, see [LICENSE](LICENSE). The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by
|
||||
@fontawesome, under [CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG.
|
||||
|
||||
27
TODO.md
Normal file
27
TODO.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# To do
|
||||
|
||||
## Trim the state database
|
||||
|
||||
From an audit of the database layer and a read-only copy of production on 2026-09-12. The data
|
||||
itself was clean: no leftover tables or indexes, 47 free pages, one stray `entry_state` row.
|
||||
Check each against the code before cutting it.
|
||||
|
||||
- [x] **`migrate()`'s add list.** All eight columns arrived in 0.2.0, and 0.5.0 only supports
|
||||
upgrades from 0.3.0 on. Drop the list and its loop; keep the `retired` drop loop, which a
|
||||
database coming from 0.4.0 still needs. (`src/db.rs`)
|
||||
- [x] **`Db::subscribed_feed_ids`.** No callers; its doc says the scanner walks it, and it does not.
|
||||
(`src/db.rs`)
|
||||
- [x] **`Db::subscriber_count`.** One caller, which only asks whether it is above zero:
|
||||
`subscriber_counts()?.contains_key(&id)`. (`src/db.rs`, `src/web.rs`)
|
||||
- [x] **The `created` columns** on `users`, `subscriptions` and `sessions`. Written on every insert,
|
||||
never read. Add them to `retired` and drop them from the inserts. (`src/db.rs`)
|
||||
- [x] **`Managed.orphaned`.** Selected by `managed_feeds()` on every call and never read;
|
||||
`FeedSummary.orphaned` is what the UI uses. (`src/db.rs`)
|
||||
|
||||
## Popular
|
||||
|
||||
- [x] **Popular lists feeds the way Directory does**: the feeds inside an OPML or a Patreon
|
||||
creator, never the collection itself. (`src/web.rs`, `web/index.html`)
|
||||
|
||||
After these: `cargo test`, `node tests/page-smoke.js`, `npx playwright test`. Copy `state.db`
|
||||
aside before deploying: `migrate()` drops columns on the first start.
|
||||
@@ -1,7 +0,0 @@
|
||||
[Unit]
|
||||
Description=ipx feed scan (one shot)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=%h/.cargo/bin/ipx fetch
|
||||
Environment=IPX_LOG=ipx=info
|
||||
@@ -1,18 +0,0 @@
|
||||
# User unit: install to ~/.config/systemd/user/ipx.service, then
|
||||
# systemctl --user enable --now ipx
|
||||
# The socket lands in $XDG_RUNTIME_DIR/ipx.sock by default, so a UI running as the
|
||||
# same user can attach without extra configuration.
|
||||
[Unit]
|
||||
Description=ipx podcatcher
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=%h/.cargo/bin/ipx daemon
|
||||
Restart=on-failure
|
||||
RestartSec=30
|
||||
Environment=IPX_LOG=ipx=info
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -1,16 +0,0 @@
|
||||
# Alternative to the daemon: a periodic one-shot scan, closer to how the original
|
||||
# iPodderX agent was driven. Use this OR ipx.service, not both -- with no daemon
|
||||
# running there is no socket, so a UI cannot attach.
|
||||
#
|
||||
# Install ipx-scan.service and ipx.timer to ~/.config/systemd/user/, then
|
||||
# systemctl --user enable --now ipx.timer
|
||||
[Unit]
|
||||
Description=Periodic ipx feed scan
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
OnUnitActiveSec=1h
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
146
docs/architecture.md
Normal file
146
docs/architecture.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# How it works
|
||||
|
||||
One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a Unix-socket
|
||||
control server, and the web UI. Everything else is a CLI that either does the work itself or hands
|
||||
it to a running daemon.
|
||||
|
||||
## Modules
|
||||
|
||||
| File | Responsibility | What it replaced in the Python |
|
||||
|---|---|---|
|
||||
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
|
||||
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
|
||||
| `src/db.rs` | SQLite schema, migrations, every query | `.ipxd` plists, `history.dat`, `qmcache.dat` |
|
||||
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
|
||||
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
|
||||
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
|
||||
| `src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` |
|
||||
| `src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout |
|
||||
| `src/auth.rs` | Argon2id hashing, session tokens, header names | — |
|
||||
| `src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — |
|
||||
| `src/logbuf.rs` | Ring buffer behind the UI's Log view | — |
|
||||
| `web/index.html` | The whole front end, `include_str!`d into the binary | — |
|
||||
|
||||
The page is compiled in, so **editing `web/index.html` needs a rebuild**.
|
||||
|
||||
## A scan
|
||||
|
||||
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
|
||||
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
|
||||
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
|
||||
re-derived into the database each scan, never written to config.toml. A Patreon creator link
|
||||
(a token, no `show=`) with more than one show is treated the same way, before any fetch: its
|
||||
shows come from Patreon's web API and each becomes a derived feed.
|
||||
4. Record entries. A changed title or description flips the item back to unread.
|
||||
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
|
||||
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
|
||||
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
|
||||
a reason. What a filter skipped is judged again every scan, so a change of settings brings it
|
||||
back. A feed in a group takes your settings on the group for anything you have not set on it.
|
||||
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
|
||||
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
|
||||
or an error page — and is deleted.
|
||||
|
||||
## Data model
|
||||
|
||||
```
|
||||
feeds id, url, title, image, etag, last_modified, last_checked, ttl_mins,
|
||||
last_error, orphaned, group_id, managed
|
||||
entries feed_id, guid, title, link, published, description, first_seen,
|
||||
image, duration, episode, season PK (feed_id, guid)
|
||||
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,
|
||||
bytes_done, downloaded_at, last_error
|
||||
users id, name, pass_hash, is_admin, created, last_login
|
||||
sessions token, user_id, seen
|
||||
subscriptions user_id, feed_id, keywords, auto_download, allow_explicit,
|
||||
max_new_per_check PK (user_id, feed_id)
|
||||
entry_state user_id, feed_id, guid, read, flagged, position
|
||||
PK (user_id, feed_id, guid)
|
||||
```
|
||||
|
||||
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
|
||||
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
|
||||
older database.
|
||||
|
||||
Schema changes: add the table or column to `SCHEMA`. `CREATE TABLE IF NOT EXISTS` leaves a table
|
||||
that already exists alone, so a new column on one also goes in `migrate()`'s `wanted` list, and a
|
||||
retired one in its `retired` list; both are checked with `PRAGMA table_info`. Columns from before
|
||||
0.3.0, the oldest version an upgrade may start from, need no entry. `Db::memory()` runs the same
|
||||
path as `Db::open`, so a migration cannot pass the tests while missing in production.
|
||||
|
||||
## Control socket
|
||||
|
||||
Newline-delimited JSON, both directions, over `$XDG_RUNTIME_DIR/ipx.sock`.
|
||||
|
||||
```sh
|
||||
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
|
||||
{"ev":"feed_start","feed":"atp"}
|
||||
{"ev":"progress","feed":"atp","enclosure":42,"file":"ep1.mp3","done":8192,"total":3000000}
|
||||
{"ev":"download_done","feed":"atp","enclosure":42,"path":"…","bytes":3000000}
|
||||
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
|
||||
{"ev":"scan_done","feeds":1}
|
||||
```
|
||||
|
||||
**Commands** — `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `download`
|
||||
(`enclosure`), `status`.
|
||||
|
||||
**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
|
||||
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
|
||||
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
|
||||
there. Commands run one at a time, in the order they arrive, except `status`: the socket answers it
|
||||
straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and
|
||||
answers only the client that asked, since `status` would end any other client's session.
|
||||
|
||||
Progress carries the enclosure id, without which a UI cannot tell one download from another and
|
||||
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
|
||||
so a client attached to a busy daemon also sees that daemon's other work.
|
||||
|
||||
Inside the process the same events go over a `tokio::broadcast`; commands arrive on an `mpsc` and
|
||||
are handled by a single worker, so nothing races over the same download. Shutdown is a `watch`
|
||||
channel raced *inside* each job — `tokio::select!` only races branches at the point of selection,
|
||||
so a long download had to be able to notice the signal itself.
|
||||
|
||||
## HTTP API
|
||||
|
||||
Everything below `/api` needs a signed-in user; the browser gets a redirect to `/login`, anything
|
||||
else a `401`.
|
||||
|
||||
| Route | |
|
||||
|---|---|
|
||||
| `GET /` | the app |
|
||||
| `GET /login`, `POST /api/login`, `POST /api/logout`, `GET /api/me` | sign-in |
|
||||
| `GET /api/feeds`, `POST /api/feeds` | your subscriptions; subscribe |
|
||||
| `PATCH /api/feeds/{id}`, `DELETE /api/feeds/{id}` | your settings or (admin) the feed's; unsubscribe |
|
||||
| `GET /api/feeds/{id}/entries` | paged, filtered, searchable, sortable (`sort` = kept, title, feed, type, size or published; `dir` = asc or desc) |
|
||||
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
|
||||
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
|
||||
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
|
||||
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, kept, position |
|
||||
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
|
||||
| `POST /api/fetch` | |
|
||||
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
|
||||
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with an OPML's feeds in place of the OPML and everyone counted (id, title, art, count, whether it is yours; never a URL, never a private feed); subscribe by id |
|
||||
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
|
||||
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |
|
||||
| `GET /api/events` | SSE, the same broadcast the socket carries |
|
||||
| `GET /api/logs` | admin-only; the ring buffer, with a sequence cursor |
|
||||
| `GET /media/{id}` | the file, with Range support so seeking works |
|
||||
|
||||
Show notes are feed-supplied HTML from an untrusted source, sanitized with `ammonia` server-side
|
||||
before they reach the page.
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
cargo test # parsing, filters, retention, schedules, SQL, per-user isolation
|
||||
node tests/page-smoke.js # the page script loads and every selector it wires at load exists
|
||||
npx playwright test # a real browser against a real daemon on fixture feeds
|
||||
```
|
||||
|
||||
The Rust tests cannot see a wrong selector, a handler that runs and does nothing, or a page that
|
||||
renders empty — which is what has actually reached users. Each Playwright case maps to a bug that
|
||||
did.
|
||||
|
||||
The suite starts its own daemon and database under `/tmp/ipx-ui-test`, wiped once per run. Tests
|
||||
share that daemon and run in order, so a test that marks something read changes what later tests
|
||||
see — make assertions that do not depend on earlier ones.
|
||||
91
docs/cli.md
Normal file
91
docs/cli.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Command line
|
||||
|
||||
```
|
||||
ipx [--config PATH] [--local] <command>
|
||||
```
|
||||
|
||||
Every command that has a wire form probes the control socket first: if a daemon is running, the
|
||||
daemon does the work and the CLI just renders the events it streams back. That is deliberate — two
|
||||
processes must never download the same thing. `--local` forces the work to happen in-process.
|
||||
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
| `ipx list` | Subscriptions and their state |
|
||||
| `ipx status` | Counts: feeds, pending, downloaded |
|
||||
| `ipx fetch [FEED] [--force]` | Scan everything, or one feed. `--force` ignores the TTL |
|
||||
| `ipx add <url> [--folder X] [--keywords a,b]` | Subscribe; the id comes from the feed title |
|
||||
| `ipx rm <feed>` | Unsubscribe; downloads and history are kept |
|
||||
| `ipx import <file.opml>` / `ipx export <file.opml>` | Move subscriptions in or out. Import subscribes the first admin, as the shared web token does; in the web UI it subscribes whoever is signed in |
|
||||
| `ipx reap [--dry-run]` | Run retention now |
|
||||
| `ipx user <add\|list\|passwd\|rm>` | Accounts for the web UI |
|
||||
| `ipx daemon [--web ADDR]` | Scheduler, control socket and web UI |
|
||||
|
||||
## Accounts
|
||||
|
||||
Passwords are read from **stdin**, so they miss the shell history and any `ps` listing.
|
||||
|
||||
```sh
|
||||
echo -n 'a good password' | ipx user add ray # local account
|
||||
ipx user add ray@example.com --no-password # signs in through the proxy only
|
||||
echo -n 'a good password' | ipx user passwd admin # change a password
|
||||
ipx user list # who exists, and how each signs in
|
||||
ipx user rm sam # account, subscriptions and read state
|
||||
```
|
||||
|
||||
The first account created is an admin; later ones are ordinary users. A database with no accounts
|
||||
at all gets **admin / ipodderx** on the next daemon start, announced in the log — change it.
|
||||
|
||||
To avoid even the command line, read it interactively:
|
||||
|
||||
```sh
|
||||
read -s PW && echo -n "$PW" | ipx user passwd admin
|
||||
```
|
||||
|
||||
## Scanning
|
||||
|
||||
```sh
|
||||
ipx fetch # everything due
|
||||
ipx fetch atp --force # one feed, ignoring its TTL and schedule
|
||||
```
|
||||
|
||||
A scan: conditional GET (`If-None-Match` / `If-Modified-Since`), parse, record new entries, apply
|
||||
the filters, then download up to the per-scan cap, newest first. A feed nothing has changed in
|
||||
answers `304` and costs one request.
|
||||
|
||||
## Retention
|
||||
|
||||
```sh
|
||||
ipx reap --dry-run # what would go, oldest first
|
||||
ipx reap # actually delete
|
||||
```
|
||||
|
||||
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
|
||||
with no file are pruned from the database. **An item anyone kept keeps its file**, and one only counts
|
||||
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
|
||||
stops the next scan fetching it again.
|
||||
|
||||
## The daemon
|
||||
|
||||
```sh
|
||||
ipx daemon # scheduler + socket + web UI
|
||||
ipx daemon --web 0.0.0.0:8099 # override the configured bind for one run
|
||||
```
|
||||
|
||||
One daemon per socket; a second refuses to start rather than fight over the database. It shuts down
|
||||
cleanly on SIGTERM, including mid-download.
|
||||
|
||||
To kill it, match the binary exactly:
|
||||
|
||||
```sh
|
||||
pkill -x ipx
|
||||
```
|
||||
|
||||
`pkill -f ipx` matches the shell running the command too, and kills your own session.
|
||||
|
||||
## Talking to it directly
|
||||
|
||||
```sh
|
||||
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
|
||||
```
|
||||
|
||||
See [architecture.md](architecture.md#control-socket) for the protocol.
|
||||
120
docs/configuration.md
Normal file
120
docs/configuration.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Configuration
|
||||
|
||||
One TOML file, read at startup and re-read whenever the web UI writes to it — most changes take
|
||||
effect without a restart. Default location `$XDG_CONFIG_HOME/ipx/config.toml`
|
||||
(`~/.config/ipx/config.toml`), overridden with `--config` or `$IPX_CONFIG`.
|
||||
|
||||
| What | Where | Override |
|
||||
|---|---|---|
|
||||
| Config | `~/.config/ipx/config.toml` | `--config`, `$IPX_CONFIG` |
|
||||
| Database | `~/.local/share/ipx/state.db` | `$IPX_DATA_DIR` |
|
||||
| Control socket | `$XDG_RUNTIME_DIR/ipx.sock` | `[general] socket` |
|
||||
| Downloads | `[general] download_dir` | — |
|
||||
|
||||
`~` is expanded in paths. The database is SQLite in WAL mode; back it up by copying `state.db`
|
||||
while the daemon is stopped, or with `sqlite3 state.db .backup`.
|
||||
|
||||
## `[general]`
|
||||
|
||||
```toml
|
||||
[general]
|
||||
download_dir = "~/Podcasts"
|
||||
socket = "/run/user/1000/ipx.sock"
|
||||
schedule = "every 1h" # "every 30m", "every 4h", "2d", "90" (minutes)
|
||||
organize = "feed" # "feed" | "date"
|
||||
max_total_gb = 50 # 0 = unlimited
|
||||
max_age_days = 30 # 0 = keep forever
|
||||
max_new_per_check = 3 # per feed, per scan. 0 = unlimited
|
||||
media_types = ["audio", "video"]
|
||||
```
|
||||
|
||||
* **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to
|
||||
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
|
||||
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
|
||||
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
|
||||
pad. Kept items are never deleted, and a file only counts as read once every subscriber has
|
||||
read it. `0` disables it entirely.
|
||||
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
|
||||
Kept ones stay. `0` disables it.
|
||||
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
|
||||
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
|
||||
fetched 216 files and 22 GB in one scan.
|
||||
* **`media_types`** — top-level MIME types taken automatically. Anything else is still listed and
|
||||
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
|
||||
without this the disk fills with artwork. Empty takes everything.
|
||||
|
||||
## `[torrent]`
|
||||
|
||||
```toml
|
||||
[torrent]
|
||||
enabled = true
|
||||
seed_ratio = 1.0 # stop seeding at this ratio ...
|
||||
seed_time_mins = 60 # ... or after this long, whichever comes first
|
||||
port_range = "6881-6889"
|
||||
stall_mins = 30 # give up on a torrent making no progress
|
||||
```
|
||||
|
||||
A `.torrent` body is handed to the torrent path whatever MIME type it was advertised as. Torrents
|
||||
run on their own tasks (two at a time) so a slow swarm never blocks a scan.
|
||||
|
||||
## `[web]`
|
||||
|
||||
```toml
|
||||
[web]
|
||||
enabled = true
|
||||
bind = "0.0.0.0:8099" # 127.0.0.1:8080 by default
|
||||
token = "" # generated and saved on first run
|
||||
trusted_header = "" # e.g. "Cf-Access-Authenticated-User-Email"
|
||||
trusted_proxies = ["127.0.0.1", "::1"]
|
||||
auto_create_users = true
|
||||
sign_out_url = "" # e.g. "/cdn-cgi/access/logout"
|
||||
session_days = 30
|
||||
```
|
||||
|
||||
* **`token`** — the shared secret, which signs in as the **admin**. `?token=…` sets a cookie, so
|
||||
you paste it once per browser. It is what the Docker healthcheck and any scripts use.
|
||||
* **`trusted_header`** — a header naming the signed-in user, set by whatever fronts ipx. Empty
|
||||
disables that path. See [sso.md](sso.md).
|
||||
* **`trusted_proxies`** — addresses allowed to assert that header, and the entire security boundary
|
||||
for it. Name the proxy, never a subnet.
|
||||
* **`auto_create_users`** — create an account the first time the proxy vouches for a new name.
|
||||
* **`sign_out_url`** — where Sign out sends someone the proxy signed in: the proxy's own sign-out,
|
||||
`/cdn-cgi/access/logout` behind Cloudflare Access. Empty sends them to the sign-in page, where
|
||||
the proxy signs them straight back in.
|
||||
* **`session_days`** — sign a session out after this long without a request.
|
||||
|
||||
It is plain HTTP. On a LAN bind everything crosses the network in the clear — and a feed URL can
|
||||
itself carry a credential. Put TLS in front of it if that matters.
|
||||
|
||||
## `[feeds.<id>]`
|
||||
|
||||
The table key is the feed id: stable, human-readable, and used in paths and the API. `ipx add`
|
||||
derives it from the feed title.
|
||||
|
||||
```toml
|
||||
[feeds.atp]
|
||||
url = "https://atp.fm/rss"
|
||||
folder = "Accidental Tech Podcast" # default: the feed title
|
||||
schedule = "every 6h" # overrides [general] for this feed
|
||||
media_types = ["audio"] # overrides [general] for this feed
|
||||
username = "ray" # HTTP basic auth
|
||||
password_env = "IPX_ATP_PASS" # preferred over a literal `password`
|
||||
```
|
||||
|
||||
With more than one account, **`keywords`, `auto_download`, `allow_explicit` and
|
||||
`max_new_per_check` live on each person's subscription in the database**, not here — the values in
|
||||
config.toml are the fallback for a feed nobody has claimed. The keys above describe the feed itself
|
||||
and are the same for everyone. See [users.md](users.md).
|
||||
|
||||
Feeds derived from a subscribed OPML are **not** written here: the OPML is the source of truth and
|
||||
they are re-derived on every scan. Editing one in the UI promotes it to a real config entry.
|
||||
|
||||
## Environment
|
||||
|
||||
| Variable | Effect |
|
||||
|---|---|
|
||||
| `IPX_CONFIG` | Config file path |
|
||||
| `IPX_DATA_DIR` | Directory holding `state.db` |
|
||||
| `IPX_LOG` | What reaches stderr (`ipx=debug`, `ipx::scan=debug`, …) |
|
||||
| `IPX_UI_LOG` | What the in-process log buffer captures for the UI's Log view |
|
||||
| `http_proxy` / `https_proxy` | Honoured for feed and enclosure fetches |
|
||||
@@ -1,58 +1,513 @@
|
||||
# Progress
|
||||
# How ipx was built
|
||||
|
||||
Running record of what has actually landed. Newest entry first.
|
||||
The full design and step list live in the plan file at
|
||||
`/config/.claude/plans/i-want-to-create-playful-quiche.md`.
|
||||
The long-form record, newest first: what landed each day, what was wrong before, and what it cost
|
||||
to find out. [CHANGELOG.md](../CHANGELOG.md) has the short version, by release; this is where the
|
||||
reasoning lives. New write-ups go at the top.
|
||||
|
||||
## Build order
|
||||
See [README.md](../README.md) for what the thing is.
|
||||
|
||||
- [x] **1. Repo skeleton** — git init (`main`), `cargo init --name ipx`, deps pinned, LICENSE,
|
||||
README, this file.
|
||||
- [x] **2. `config.rs` + `db.rs`** — TOML config structs + SQLite schema.
|
||||
- [x] **3. `feed.rs`** — conditional GET, RSS-then-Atom parse, persist entries.
|
||||
- [x] **4. `download.rs`** — downloads, filters, dedupe.
|
||||
- [x] **5. `retention.rs`** — oldest-first quota + age reaper.
|
||||
- [x] **6. `ipc.rs` + daemon** — UDS JSON-lines server, TTL scheduler, CLI-proxies-to-daemon.
|
||||
- [x] **7. `torrent.rs`** — librqbit, seed to ratio/time, stall abort. (swarm download unverified —
|
||||
see the step 7 entry)
|
||||
- [x] **8. OPML + polish** — import/export, add/rm/status, tracing setup, systemd units, README.
|
||||
## 2026-09-12 — Healthy while busy
|
||||
|
||||
### Phase 2 — web front end
|
||||
After a deploy the container sat at "starting" for a minute, and Docker's health log showed two
|
||||
`ipx status` probes exceeding their 5-second timeout. The daemon's own log explained it. The first
|
||||
scan after the start fetched 23 feeds, from 14:10:41 to 14:11:35, and both probes' `status`
|
||||
commands waited in the job queue behind it; they were answered together at 14:11:35, straight after
|
||||
`scan_done`. The worker runs one job at a time and `status` was one of its jobs, so any scan or
|
||||
download longer than about a minute and a half, three 30-second probes, would have had Docker call
|
||||
a working daemon unhealthy.
|
||||
|
||||
Decided with Ray: axum serving plain HTML/JS (no WASM toolchain), running **inside the daemon**
|
||||
process so it reads SQLite and the event bus directly, LAN-bindable with a shared token.
|
||||
The socket now answers `status` itself, from two short queries, and only real work goes through the
|
||||
queue. The trade is that healthy now means the daemon answers on its socket and can read its
|
||||
database; a worker stuck on one job would still pass. Asking a daemon that downloads hour-long
|
||||
podcasts to be idle within five seconds was never a fair test of whether it was alive. A test holds
|
||||
the queue full and checks `status` still comes back.
|
||||
|
||||
- [x] **9. Config hot-reload + web skeleton.** `Ctx.cfg` becomes `RwLock<Arc<Config>>` so the UI can
|
||||
edit feeds without a daemon restart. `[web]` config section (enabled/bind/token, token
|
||||
auto-generated and saved on first run). axum server started by `ipx daemon`, token checked by
|
||||
middleware, `?token=` sets a cookie so `<audio>` requests authenticate too.
|
||||
*Done when:* `ipx daemon` serves a page on the configured bind, and a wrong token gets 401.
|
||||
- [x] **10. Browsing.** `/api/feeds`, `/api/feeds/:id/entries`, entry detail. Descriptions are
|
||||
untrusted feed HTML — sanitized with `ammonia` before they reach the page.
|
||||
*Done when:* the Glass Cannon feed's 131 entries browse and read correctly.
|
||||
- [x] **11. Media actions.** Range-request audio streaming (`tower-http` ServeFile) so seeking
|
||||
works, download-on-demand for a pending enclosure, delete a file, mark read/flagged.
|
||||
*Done when:* an episode plays and seeks in a browser, and delete reaps the row.
|
||||
- [x] **12. Feed configuration.** Add/remove feeds and edit folder, keywords, allow_explicit,
|
||||
auto_download, max_new_per_check from the UI, written back to config.toml and hot-reloaded.
|
||||
*Done when:* flipping allow_explicit in the UI takes effect on the next scan with no restart.
|
||||
- [x] **13. Live progress + polish.** SSE from the existing broadcast bus so downloads show live.
|
||||
README section, screenshot-free usage notes.
|
||||
*Done when:* starting a fetch from the UI shows progress advancing without a reload.
|
||||
The first version broadcast the answer, as the queued one had been. Timing `status` during a forced
|
||||
scan in production showed the catch: `status` is a terminal event, so the `ipx fetch` watching that
|
||||
scan stopped reading at the first probe and printed the status line as its last, while the scan
|
||||
carried on. When `status` waited behind the scan it could never arrive first, so this had never
|
||||
shown. The answer now goes only to the client that asked, and the test checks that another client
|
||||
hears nothing.
|
||||
|
||||
Note: `read`/`flagged` finally get a writer here. Retention orders by them (see the step 5 entry),
|
||||
and until now nothing set them.
|
||||
## 2026-09-12 — Signing in through Authentik, for real
|
||||
|
||||
## Smoke tests
|
||||
Ray could not get Authentik's sign-in to reach ipx, following `docs/sso.md`, which had been written
|
||||
without ever being tried. Looking at the Cloudflare account through its API showed that side was
|
||||
already complete. Authentik is Zero Trust's OpenID Connect identity provider; the Access application
|
||||
`ipodderx` allows only it and a list of five addresses; the tunnel `rays-unraid` routes
|
||||
`ipodderx.sdf1.net` to `192.168.1.130:8099`; DNS is a proxied CNAME to the tunnel. Access's log
|
||||
showed `rays@sdf1.net` signing in through it. Nothing on Cloudflare was changed, so no other site
|
||||
was touched.
|
||||
|
||||
1. `ipx add <feed>` + `ipx fetch` → file in `download_dir/<Show>/`, row in `enclosures`.
|
||||
2. `ipx fetch` again → no re-download, feed skipped for TTL.
|
||||
3. `ipx daemon &` + `nc -U $XDG_RUNTIME_DIR/ipx.sock`, send `{"cmd":"fetch"}` → JSON events;
|
||||
a concurrent `ipx fetch` proxies to the daemon instead of downloading in parallel.
|
||||
4. Delete a downloaded file by hand, `ipx fetch` → NOT re-downloaded.
|
||||
5. Torrent enclosure → downloads, moves, stops seeding at the configured ratio/time.
|
||||
6. `ipx reap --dry-run` under quota pressure → oldest-first hit list; real run flips rows to
|
||||
`reaped`.
|
||||
The gaps were all at ipx's end: `trusted_header` was empty, `trusted_proxies` held only loopback,
|
||||
and the account was called `rays` while the header carries `rays@sdf1.net`.
|
||||
|
||||
Finding the address to trust took the most time. The page said `127.0.0.1`, but `cloudflared` runs in
|
||||
its own container and reaches ipx through the host's published port. ipx logs no peer addresses, so
|
||||
the address was read from `/proc/net/tcp` inside the ipx container: `192.168.16.1`, the gateway of
|
||||
`content_default`, where Docker's masquerade puts traffic crossing from another bridge. A request
|
||||
from Tower's own shell arrived as `192.168.1.130` instead, and a throwaway `busybox` on the default
|
||||
bridge as `192.168.16.1`: the first was refused with the header, the second believed. LAN machines
|
||||
keep their own addresses, since Docker forwards published ports with iptables (the userland proxy
|
||||
only handles loopback).
|
||||
|
||||
Every change, in order, with how to undo it:
|
||||
|
||||
1. **Code**, commit `586d2c0`: `ipx user rename`, deployed. Revert the commit and redeploy to
|
||||
remove it; nothing depends on it once used.
|
||||
2. **Account**: `docker exec iPodderX ipx user rename rays rays@sdf1.net`. Same id, so its feeds,
|
||||
read state, password and admin rights stayed. Undo: `docker exec iPodderX ipx user rename
|
||||
rays@sdf1.net rays`. Signing in at `/login` now takes the new name.
|
||||
3. **Config**, `/mnt/fast/appdata/ipodderx/config.toml`, `[web]`: `trusted_header` from `""` to
|
||||
`"Cf-Access-Authenticated-User-Email"`, and `"192.168.16.1"` added to `trusted_proxies`. The
|
||||
file as it was is `config.toml.2026-09-12-sso.bak` beside it. Undo: copy the backup back and
|
||||
`docker compose -f /mnt/fast/arcane/projects/content/compose.yaml restart ipodderx`.
|
||||
4. **Cloudflare, Docker networks and other containers**: unchanged. The `busybox` test container
|
||||
was removed when it exited, and its image afterwards.
|
||||
5. **Authentik**, later the same day, because ipodderx had no tile in its library while Outline
|
||||
did: a bookmark application `ipodderx` (pk `5854a98e-816a-4c4f-9f27-63e69dc29d1d`), made
|
||||
through the API with a token of Ray's. No provider and no policy bindings, like Outline's, the
|
||||
iPodderX icon, and a link to `https://ipodderx.sdf1.net`. It changes nothing about who can sign
|
||||
in. Undo: delete it under Applications → Applications, or
|
||||
`DELETE /api/v3/core/applications/ipodderx/`.
|
||||
6. **Signing out**, later again. Sign out landed on ipx's password page while Access still vouched
|
||||
for Ray, so it signed nothing out, and the page looked like the wrong login. Cloudflare's
|
||||
`/cdn-cgi/access/logout` ends the Access session for every Access application at once (there is
|
||||
no per-application sign-out, and it takes no redirect), and Authentik's end-session only ends
|
||||
one application's session unless single logout is set up there. Ray chose Access's sign-out. New
|
||||
`[web] sign_out_url`, set to `/cdn-cgi/access/logout` in production (the file as it was is
|
||||
`config.toml.2026-09-12-signout.bak`), and `/login` now sends anyone the proxy vouches for on to
|
||||
`/`. Undo: take the key out and restart; the code does nothing without it.
|
||||
|
||||
What the address trusts is any container on Tower that connects through the host's port, not only
|
||||
`cloudflared`. Verifying Cloudflare's signed `Cf-Access-Jwt-Assertion` would remove that, and is
|
||||
the upgrade if it matters.
|
||||
|
||||
## 2026-09-12 — Trimming the state database
|
||||
|
||||
An audit of the database layer, with a read-only copy of production to check it against. The
|
||||
data was already clean: no tables or indexes left from older versions, 47 free pages after the
|
||||
column drops earlier the same day, and one stray `entry_state` row. The code had five things:
|
||||
|
||||
- `migrate()` still added eight columns to any table missing them. All eight shipped in 0.2.0 and
|
||||
upgrades now start from 0.3.0 at the oldest, so the list and its loop went; the `retired` drop
|
||||
list stays, since a database coming from 0.4.0 still has the old read columns.
|
||||
- `created` on `users`, `subscriptions` and `sessions` was written by every insert and read by
|
||||
nothing. They joined `retired`. The old-database test now builds all three tables, foreign keys
|
||||
included, since `DROP COLUMN` on a table that references another was the part worth proving.
|
||||
- `Db::subscribed_feed_ids` had no callers, though its doc said the scanner walked it.
|
||||
`Db::subscriber_count` had one caller asking whether it was above zero, which
|
||||
`subscriber_counts().contains_key` answers. `Managed.orphaned` was selected and never read.
|
||||
- `users.created` came back the same afternoon, with `last_login` beside it. Nothing read it, but
|
||||
when an account was made and when it last signed in is what you want to know when tidying
|
||||
accounts, and it cannot be recovered later. Both existing accounts got their creation times back
|
||||
from the backup taken before the drop, and a last sign-in from their newest session in it.
|
||||
`last_login` is kept to the hour, because the proxy vouches for every request and that would
|
||||
otherwise be a write each time.
|
||||
|
||||
## 2026-09-12 — Cutting what had outlived its reason
|
||||
|
||||
A whole-repo audit for over-engineering listed twelve things to cut, and all of them went.
|
||||
|
||||
- **Upgrades from before accounts.** `migrate_opml_children` moved OPML feeds that old versions
|
||||
wrote into `config.toml` out to the database, and ran at every daemon start to do nothing after
|
||||
the first. Production ran it in 0.3.0; anything older has to pass through 0.4.0.
|
||||
- **Half of the adoption, and not the other half.** The audit called `adopt_existing_library` a
|
||||
one-time migration and it was cut whole. It did two jobs: copy the old read state into
|
||||
`entry_state`, which was dead, and subscribe the first admin to the whole catalogue while nobody
|
||||
subscribed to anything, which is how a fresh install's first account gets `config.toml`'s feeds.
|
||||
The browser suite caught it at once, signing in to an empty sidebar; `cargo test` had no idea.
|
||||
The second job is back as `adopt_catalogue`, with a unit test of its own.
|
||||
- **The dead `entries` columns.** `read`, `flagged` and `position` moved to `entry_state` with
|
||||
accounts. The adoption's copy was their last reader, but `record_entry` still wrote them, and
|
||||
still reset `read` when a title changed, which nothing looked at. Two bugs came from queries
|
||||
reading them. `migrate()` now drops them from an existing database (SQLite has had `DROP COLUMN`
|
||||
since 3.35), and a test builds an old table to prove it. On production each drop rewrote the
|
||||
66 MB `entries` table, about four seconds apiece, so the first start took thirteen. An
|
||||
`ipx status` run in that window failed with `migrating schema`: every `ipx` command migrates when
|
||||
it opens the database, and it collided with the daemon doing the same. A failed `ALTER TABLE`
|
||||
changes nothing, and the database had been copied to `backup/` first anyway.
|
||||
- **`interval_mins`**, which `schedule` replaced. An old config that still has the key loads; the
|
||||
key is ignored, and the config test carries it to keep that true.
|
||||
- **Three dependencies.** `infer` was only asked whether a file is a torrent, and the check after
|
||||
it already looked for `d8:announce`, which is what `infer` looks for. `dirs` was three lookups of
|
||||
`XDG_CONFIG_HOME`, `XDG_DATA_HOME` and `HOME`. `tokio-stream` wrapped the broadcast receiver for
|
||||
the event stream; `futures_util::stream::unfold` does the same, lagging clients included.
|
||||
- **Two token generators.** The web token came from a copy of the session-token code, with a
|
||||
clock fallback on top. It uses `auth::new_session_token` now, and is 64 characters.
|
||||
- **The icon inlined four times**, 23 KB of base64 each, into both pages. It is `/icon.png` now,
|
||||
outside the sign-in wall with `/login`, since the sign-in page shows it.
|
||||
- Also: the `contrib/` systemd units from before the container, `Db::entries` and
|
||||
`Db::count_entries` that only the tests called, three `logbuf` visitors that repeated the trait's
|
||||
defaults, and unused state, a helper and dead CSS in the page.
|
||||
|
||||
## 2026-09-11 — A design pass on the web UI
|
||||
|
||||
A review against screenshots of every view in all three themes found that Dark and Light read as a
|
||||
generic dark dashboard: one pale blue did every job, most labels were bold, and nothing led. The
|
||||
list it produced, in `TODO.md`, was worked through in one go. What was worth knowing:
|
||||
|
||||
- **Amber means new.** Badges, unread dots and download bars take the icon's EQ amber; blue is left
|
||||
for the primary action and links. Light's amber was `#b06f10`, which gives white text 4.1:1,
|
||||
short of AA for 11 px bold. It is `#9a5f0a` now, 5.2:1.
|
||||
- **EQ bars mark what is playing.** Three `<i>` bars stand at 60, 100 and 40 % and animate only
|
||||
while `body.playing` is set. The first version left the animation on but paused, expecting each
|
||||
bar to hold a different frame. The frames it held were within a pixel of each other, and on
|
||||
screen the bars read as three dots. Under reduced motion one rule drops every animation and
|
||||
transition, which leaves the bars standing.
|
||||
- **The focus ring was clipped.** `.tgroup` and `#topbar` both set `overflow:hidden`, so a ring
|
||||
drawn outside a toolbar button was cut off. Rings inside clipping parents are inset instead.
|
||||
- **The feed list could not be used from the keyboard at all.** Rows were `<div>`s and the triangle
|
||||
a `<span>`, so Tab went from the feed filter to Sign out. Rows now take focus, the triangle is a
|
||||
`<button aria-expanded>`, and `renderFeeds` puts focus back on the same row after redrawing,
|
||||
since every live update replaces every row. The list's own key handler stops Space and the
|
||||
arrows from reaching the player's shortcuts on the document.
|
||||
- **The triangle hangs in the margin.** Every row used to reserve an 18 px slot for it, pushing a
|
||||
hundred feeds 28 px right of the places above for the sake of two folders. It is now absolutely
|
||||
placed in the row's left padding, the full height of the row, so a near miss no longer opens
|
||||
the folder's page.
|
||||
- **A selected tile vanished** because the initials tile and the selected row were both `--raise`.
|
||||
Tiles now mix their tint into `--bg`, which no row uses.
|
||||
- **The Files pane hides itself** with `#split:has(>#files[hidden])`, which collapses its column.
|
||||
The phone layout already hides the pane, and a zero-width extra track there is harmless.
|
||||
- **Flagged became Kept** in the tab, and in the server's refusal to delete a file someone else
|
||||
kept. The filter value and the column stay `flagged`; renaming those buys nothing.
|
||||
|
||||
## 2026-09-11 — A Patreon creator is a list of shows
|
||||
|
||||
Ray asked whether ipx could sync with Patreon. Not in full. The documented API (v2, the
|
||||
`identity.memberships` scope) lists the creators you back and whether each has a feed (`has_rss`),
|
||||
but no resource carries the `auth` token that makes a feed URL work. That token only comes from the
|
||||
creator's page. It is also one per membership, not one per account: techpod's differs from Glass
|
||||
Cannon's, so no single token finds everything you back.
|
||||
|
||||
What does work is one creator at a time, which is what Ray wanted for Glass Cannon and its 33 shows:
|
||||
|
||||
- `patreon.com/rss?auth=<token>`, with no creator named, returns that token's creator. Its self link,
|
||||
about 660 bytes in, gives the campaign by number (`/rss/369921`). Patreon ignores `Range` here, so
|
||||
ipx reads the stream until the number appears and hangs up, instead of taking all 2.8 MB.
|
||||
- A show's `show=` number is a Patreon collection. Asked anonymously, the collection listing
|
||||
(`/api/collection?filter[campaign_id]=`) and a post's `collections` both hide patron-only ones: you
|
||||
get "FAQ". `/api/campaigns/<id>?include=shows` lists every show, anonymously, in one response.
|
||||
- Every spelling works: `rss/glasscannon?auth=…&show=N`, `rss/369921?…` and `rss?auth=…&show=N` all
|
||||
serve the same 131 items. Enclosure URLs are the same in the creator feed and the show feed, and
|
||||
stable between fetches.
|
||||
|
||||
That last point shaped the design. `enclosures.url` is unique, so whichever feed is scanned first owns
|
||||
the file. The first cut only asked a creator for its shows while it had no entries of its own, so that a
|
||||
creator already read as a plain feed, holding every show's episodes, would never be split into shows
|
||||
that came up empty. Within the hour that was the wrong call: Glass Cannon had gone into production on
|
||||
the build before this one, been read as one feed of 2,385 items, and the rule kept it that way. Finding
|
||||
anything in that heap was the problem Ray wanted solved.
|
||||
|
||||
So a creator with more than one show is always a group, run through the same sync as an OPML
|
||||
(`sync_group`, split out of `sync_opml`). When it becomes one, its items are cleared and each show
|
||||
takes over the enclosures the creator holds as the show lists them (`Db::adopt`), downloaded files and
|
||||
everyone's read state included. One show leaves it a plain feed, which is what techpod already was. If
|
||||
the shows cannot be listed, a creator already split fails the scan rather than being read as one heap;
|
||||
one that never was is read as one feed until they can be. An answer without a `shows` list is an error,
|
||||
not "no shows".
|
||||
|
||||
**Filter verdicts follow the settings.** Ray also reported that turning on Allow explicit and
|
||||
rescanning brought nothing back. An item was judged once, when first seen, and `skipped` was final. The
|
||||
2026-09-10 entry below saw it coming ("worth a `ipx retry <feed>` command if this bites"). It bit:
|
||||
2,166 Glass Cannon items and all 88 of Shadowdark's sat at `skipped: explicit` with the setting on.
|
||||
Every scan now runs the filters again over what they skipped (not over `torrents disabled`, which is not
|
||||
a filter's call) and requeues what they now let through. Only that direction: a queued item is never
|
||||
pulled back, because Download latest and a manual download both work by queueing.
|
||||
|
||||
**Two gaps beside it.** Add feed had no explicit box, so every new feed's first scan skipped all its
|
||||
explicit items; it has one now, stored on your subscription like the feed dialog's. And a feed inside a
|
||||
group ignored your settings on the group, though the group's dialog said they were inherited: settings
|
||||
live on each person's subscription, and nothing read the group's. `Db::subscribers` now fills what you
|
||||
have not set on the feed from your subscription to the group, and the feed list shows the same.
|
||||
|
||||
**A name that was already used.** Replaying the split on a copy of the production database left one
|
||||
show with a Supercast episode in it. "Glass Cannon Live! Ascension | Pathfinder 2E" slugs to
|
||||
`glass-cannon-live-ascension-pathfinder-2`, the id of a Supercast feed of the same show that had been
|
||||
removed. Removing a feed keeps its rows on purpose, so that re-adding it does not fetch the back
|
||||
catalogue again, but choosing a new id only checked config.toml and derived feeds. The Patreon show took
|
||||
the old id and everything still filed under it. An id is now also taken when the database has a feed by
|
||||
that id at a different URL; the same URL may still have it back, which is the re-add case.
|
||||
|
||||
Shows already added by hand are matched by token and show number, not by exact URL (`same_feed`), so a
|
||||
bare token does not add Get in the Trunk and Shadowdark a second time under another spelling.
|
||||
|
||||
The show listing is Patreon's own undocumented web API. If it changes, only finding new shows stops.
|
||||
|
||||
## 2026-09-11 — One meaning per icon, sortable columns, and one player
|
||||
|
||||
Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act
|
||||
on. The pass started from screenshots of every view and dialog, taken against the browser suite's
|
||||
own daemon and fixtures. Several things did not agree:
|
||||
|
||||
- Unsubscribe was a minus in the toolbar but an x in the feed header. There it read as "close this
|
||||
page", and x also meant Cancel in every dialog.
|
||||
- Half the dialogs still had word buttons: Log, Add feed, Users, Unsubscribe, OPML, Popular's
|
||||
Subscribe, the Copy buttons and Sign out.
|
||||
- The player bar and the folder arrow were font characters, and so was the row's "playing" marker.
|
||||
Those came out thin and varied from font to font.
|
||||
- The OPML subscription page laid out its header differently from a feed's, and All Subscriptions
|
||||
had no buttons at all.
|
||||
|
||||
The rule now is one meaning per icon. Minus unsubscribes, x closes or cancels, plus adds, subscribes
|
||||
or imports, and a dialog's confirm button carries the icon of what it does. The words go in the
|
||||
tooltip. The toolbar is grouped by what it acts on: add, unsubscribe and scan act on feeds; play,
|
||||
read and keep act on the selected item. Its read and keep buttons now show that item's state, as
|
||||
the item's own buttons already did. A browser test opens all eight dialogs and fails on any button
|
||||
without an icon and a tooltip. It also checks that the toolbar and the page use the same icon for
|
||||
the same action.
|
||||
|
||||
The pass turned up three bugs:
|
||||
|
||||
- **Escape ignored inside a dialog's text box.** The keydown handler returned early for any key
|
||||
typed in a text box, and Escape sat below that check. So Add feed, which opens with the cursor in
|
||||
its URL box, could not be closed with Escape. It showed up because the screenshot run stalled:
|
||||
the dialog stayed open, and every later click landed on its overlay until the run timed out.
|
||||
- **A white password box in the dark theme.** The input rule listed text, search and number
|
||||
fields, but not password ones.
|
||||
- **A stray "• •" under an item with no date.** Each detail carried its own trailing dot, whether
|
||||
or not the next one was there.
|
||||
|
||||
All Subscriptions gained Mark everything read, as `POST /api/read-all`. It marks the feeds from the
|
||||
person's subscriptions, which is the same set the view's query is scoped by. It asks first, since
|
||||
one click covers every feed.
|
||||
|
||||
**Sorting** happens on the server. A list arrives fifty at a time, so sorting in the page would
|
||||
only have reordered what was loaded. The column name picks one of a fixed set of SQL expressions
|
||||
(`order_sql`), so nothing the browser sends reaches the query, and an unknown name means newest
|
||||
first. Ties also fall back to newest first, so "Load more" does not skip or repeat items at a page
|
||||
boundary. A database test runs every column both ways. Size came out of the File column into its
|
||||
own, and small files show KB, since whole megabytes made every article image "0 MB".
|
||||
|
||||
**The double play.** Ray found that pressing play in the Files pane played the file twice at once.
|
||||
The pane drew its own `<audio controls>`, and its `onplay` handler also started the player bar on
|
||||
the same file. That meant two audio elements from one click. The pane now has a play button that
|
||||
hands that exact file to the player bar, the only player. `play()` now takes the file. Its check
|
||||
for resuming compares the file as well as the item, so another of an item's files starts from its
|
||||
beginning instead of carrying on with the first. A test plays from the pane and checks the page
|
||||
has one `<audio>` element.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — The original's layout
|
||||
|
||||
Ray pointed at a screenshot of the Mac app (techpp.com, 2012) and asked for its panes, its
|
||||
toolbar, and a Directory that lives in the feed list rather than behind a button. What it had, and
|
||||
what ipx now does:
|
||||
|
||||
- A toolbar across the window: subscribe and unsubscribe, play, flag, refresh, and a search box
|
||||
scoped to the feed on show. ipx's has the same groups, acting on the selected feed and item, with
|
||||
Settings and Log at the right end, for admins only.
|
||||
- A source list opening with Directory, Playlist Builder and All Subscriptions above the feeds.
|
||||
ipx has Directory, Popular and All Subscriptions there, opening in the main pane. Playlist
|
||||
Builder is left out, since nothing here builds playlists.
|
||||
- The entries as a table, with a Files pane beside it and the entry below. The columns are unread,
|
||||
kept, the item, its feed, its file and when. Sorting by column is not done yet.
|
||||
- A status bar with the totals for what is on show.
|
||||
|
||||
All Subscriptions needed one new endpoint, `GET /api/entries`. It is the per-feed query with
|
||||
`feed_id = ?` swapped for the person's subscriptions. The enclosure lookup that follows it used to
|
||||
filter by feed as well; a page can now span feeds, so each file is matched to its row by feed and
|
||||
guid instead.
|
||||
|
||||
A phone has no room for a pane beside the table, so there the files follow the item's text in the
|
||||
full-screen reader, and the table drops to title and date.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Popular on this server
|
||||
|
||||
The old iPodderX had a directory of podcasts and a top-feeds list. The open-sourced engine shows how
|
||||
the list was fed. After every successful download, `iPXClass.py:1249` called `doPing`, which sent
|
||||
the enclosure URL and the feed URL, unencoded and over plain HTTP, to
|
||||
`directory.iPodderX.com/feedData/survey/files`. It was on by default, and only the Mac app's
|
||||
preferences could switch it off (`anonFeedback`). The directory itself lived on that server and in
|
||||
the Cocoa app, and both are gone.
|
||||
|
||||
ipx keeps the idea and drops the ping. Add feed lists what other accounts on this server subscribe
|
||||
to, ranked by subscriber count. The old ping would have sent a Patreon feed's `auth=` key along
|
||||
with everything else, so this is built around not doing that:
|
||||
|
||||
- A row carries an id, a title, artwork and a count. No URL reaches the page, so no key can.
|
||||
- Subscribing goes by id through `POST /api/popular/{id}`, which checks the id against the same
|
||||
list. Guessing a private feed's id gets a 400.
|
||||
- Anything that looks private is never listed: a login configured for the feed, credentials in
|
||||
the URL, or a query key containing `auth`, `token`, `key`, `secret`, `pass`, `sig`, `session`,
|
||||
`user` or `uid`. It is a heuristic, and a token hidden in the URL's path gets through. A per-feed
|
||||
`unlisted` flag is the upgrade if that happens.
|
||||
|
||||
It happened the same day. The first screenshot of the new Directory listed "Glass Cannon Live!
|
||||
Ascension (for Ray Slakinski)", a Supercast feed at `feeds.supercast.com/feeds/<key>`. Treating
|
||||
any long path segment as a key would have hidden public feeds too: acast's show ids look the
|
||||
same. So paid-feed services are named instead (Patreon, Supercast, Supporting Cast, Glow,
|
||||
Memberful), and any feed from one of them is private whatever its URL looks like. The per-feed
|
||||
flag is still the upgrade for a service not on that list.
|
||||
- Feeds from an OPML are left out. Everyone subscribed to an OPML counts every feed inside it, so
|
||||
they would bury everything anyone chose on purpose.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Import an OPML file by uploading it
|
||||
|
||||
The import screen now has a file picker as well as the paste box. The page reads the file and
|
||||
checks it looks like OPML before sending anything. If it doesn't, the page says so and clears the
|
||||
picker, and it also clears the picker after an import. The file travels as text and is never
|
||||
written to disk on the server, so there is nothing to clean up there.
|
||||
|
||||
The server now parses the OPML before touching anything, and returns `400` "that is not an OPML
|
||||
file" instead of a `500`. `subscribe_opml` takes a parsed document, so `ipx import` also refuses a
|
||||
non-OPML file by name before changing anything.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — OPML import and export are per person
|
||||
|
||||
Importing an OPML now subscribes you to every feed in it. Feeds already in the catalogue cost
|
||||
nothing, and unknown ones are added under the OPML's title. Before this, import only added URLs
|
||||
missing from `config.toml` and subscribed nobody. So importing an export from another account did
|
||||
nothing at all, and a genuinely new feed had no subscriber and was never scanned. The page said
|
||||
"Imported 0 feed(s)". `ipx import` had the same gap; it now subscribes the first admin. Both go
|
||||
through `subscribe_opml`.
|
||||
|
||||
Export now lists only your own subscriptions. It used to write out the whole catalogue to anyone
|
||||
signed in, including other people's feeds and any private URLs in them.
|
||||
|
||||
Tests now cover every way in and out:
|
||||
- Settings' Export OPML download and paste-to-import screen.
|
||||
- A round trip that imports the admin's actual export into a second account and checks both
|
||||
exports match. That round trip is exactly what failed.
|
||||
- `ipx import` and `ipx export`, run in their own scratch config and database.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — The log is admin-only
|
||||
|
||||
`GET /api/logs` now returns `403` to anyone who is not an admin, and the page hides the Log button
|
||||
from them. Before this, every signed-in person could read the whole log. That includes every
|
||||
account's name, every feed anyone subscribes to, and every failed sign-in. `/api/events` stays open
|
||||
to everyone, because it carries the scan progress each person's page shows.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Managing users from the web
|
||||
|
||||
Settings has a **Manage users…** screen for an admin. From it you can add someone, with a password
|
||||
or with none for someone the proxy signs in, tick or untick Admin, and remove an account. It is
|
||||
backed by `GET/POST /api/users` and `PATCH/DELETE /api/users/{id}`, which return `403` for anyone
|
||||
who is not an admin. The only admin cannot be demoted or removed, because nobody would then be
|
||||
able to manage accounts except from the CLI on the box. Before this, accounts could only be managed
|
||||
with `ipx user`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Unread feeds first inside an OPML
|
||||
|
||||
An OPML subscription's feeds, both in the sidebar folder and on its own page, now list the ones with
|
||||
unread items first. They were listed alphabetically, so with dozens of feeds the few with anything
|
||||
new were scattered through the list. Within each half the order is still alphabetical. The browser
|
||||
suite's fixture OPML gained a second feed, Aardvark Radio, which sorts first by name and by position,
|
||||
so the new test only passes if unread wins.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Deploying is a Docker image
|
||||
|
||||
Production moved from a hand-started daemon in code-server to the `iPodderX` container in the Arcane
|
||||
project `content`. `CLAUDE.md` now deploys by pushing to the registry at `192.168.1.130:5000` and
|
||||
recreating that one service with `docker compose`. The old instructions copied a binary over a
|
||||
process nobody supervised, so it did not come back after a reboot. Two known gaps are gone: the
|
||||
image does support accounts, and the entrypoint drops to `99:100`, so downloads are no longer
|
||||
owned by root. The README's Docker section said `docker compose up -d` builds the image and named the
|
||||
service `ipx`; the compose file pulls from the registry and the service is `ipodderx`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Documentation
|
||||
|
||||
`PROGRESS.md` became this changelog; the finished step lists moved to an appendix. The README is now
|
||||
an overview that points at `docs/`: [configuration](configuration.md) (every key, path and
|
||||
environment variable), [cli](cli.md), [users](users.md) (what several people share),
|
||||
[sso](sso.md) (refreshed for accounts and the admin-only settings), and
|
||||
[architecture](architecture.md) (modules, the scan, the schema, the socket protocol, the HTTP
|
||||
API). `CLAUDE.md` collects what someone working on the code needs to know and has already been
|
||||
learned the hard way here -- `pkill -x` not `-f`, the page being compiled in, the dead columns on
|
||||
`entries`, the Playwright worker that deleted its own database.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Pruning respects a star from anyone
|
||||
|
||||
Same defect as the reaper, one function along: `prune_entries` guarded on `entries.flagged`, which
|
||||
nothing has written since read state moved to `entry_state`. Starring a text item with no file would
|
||||
not have saved it from the age sweep. It follows the reaper's rule now, and takes orphaned read
|
||||
state with whatever it deletes.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Step D: one file, and everyone who wants it
|
||||
|
||||
The last of it, which is all about telling the truth before acting:
|
||||
|
||||
* **Delete says whose file it is.** A feed with other subscribers labels the button *Delete for
|
||||
everyone*, and the confirmation names how many people share it. The server then has the last
|
||||
word: if anyone else has starred the item or not played it yet, `DELETE /api/enclosures/{id}`
|
||||
returns **409** with the reason in plain words, and only `?force=true` goes through. So the
|
||||
rule is enforced where it matters rather than in the page that happens to be asking.
|
||||
* **A feed's header says it is shared** -- "shared with 1 other person" -- which is the answer to
|
||||
"why is there a file here I never asked for": someone else's subscription fetched it, and one
|
||||
copy serves you both.
|
||||
* Retention already respects the same rule from the entry before this one: starred by anyone keeps
|
||||
it, read by everyone releases it.
|
||||
|
||||
`others_wanting` is tested with three subscribers disagreeing, and a browser test walks the whole
|
||||
delete flow: the label, both prompts, declining the second, and the file still being there.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Retention caught up with per-user state
|
||||
|
||||
Moving read and starred into `entry_state` left `reap_candidates` reading `entries.read` and
|
||||
`entries.flagged`, which nothing writes any more: **starring stopped protecting a file**, and the
|
||||
"delete the ones already read first" ordering was dead. Harmless while the quota and age limits are
|
||||
0 -- the reaper deletes nothing at all then -- but it would have bitten the moment one was set.
|
||||
|
||||
One file serves every subscriber, so both tests are now about all of them: **anyone** starring it
|
||||
keeps it, and it only counts as read once **everyone** subscribed has read it. A file whose feed
|
||||
nobody subscribes to has nobody left to keep it, so it sorts with the read ones. Tested with two
|
||||
subscribers disagreeing.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-11 — Steps B and C: what is yours, what is everyone's
|
||||
|
||||
Read, starred and playback position moved out of `entries` into `entry_state (user_id, feed_id,
|
||||
guid, ...)`; subscriptions became rows in `subscriptions (user_id, feed_id, ...)` carrying **your**
|
||||
keywords, auto-download, explicit and per-scan limit. The feed list, unread counts, filters and
|
||||
mark-all-read are all per person now. On first start the existing library is adopted by the admin:
|
||||
2438 read/starred items and all 86 feeds, so nothing was lost.
|
||||
|
||||
The split follows from the file being shared:
|
||||
|
||||
* **Yours**: read state, starred, position, keywords, auto-download, explicit, per-scan limit,
|
||||
and which feeds you see at all.
|
||||
* **Everyone's**: the feed URL, its download folder, and when it is scanned -- there is one copy of
|
||||
a file however many people subscribe, so those describe the file, not a preference. Admin-only,
|
||||
refused with a 403 for anyone else rather than merely hidden.
|
||||
|
||||
Scanning merges the subscribers' wants, because one fetch and one file serve them all: an item is
|
||||
downloaded if **anyone** wants it (any one person's keyword set matching is enough, and one person
|
||||
taking everything removes the filter), auto-download is on if anyone has it on, and the per-scan cap
|
||||
is the largest anyone asked for. `merge_policy` is a pure function with a test covering each of
|
||||
those. Subscribing to a feed someone already has costs no second fetch and no second copy on disk;
|
||||
unsubscribing takes it off your list alone, and only when the last subscriber leaves does the feed
|
||||
stop being scanned.
|
||||
|
||||
**A test-harness bug worth naming**: Playwright imports the config in every worker, so the fixture's
|
||||
`prepare()` ran again mid-run and deleted the data directory out from under the daemon. The daemon
|
||||
kept serving from the unlinked inode while the CLI and any query opened a fresh empty database at
|
||||
the same path -- which looked exactly like sign-in being broken. Only the launching process wipes
|
||||
now (a worker has `TEST_WORKER_INDEX`).
|
||||
|
||||
---
|
||||
|
||||
@@ -107,34 +562,6 @@ who is signed in above the sidebar footer, with a sign-out, and a 401 sends the
|
||||
Nothing is per-user *yet*: everyone still sees the same feeds and read state. That is step B.
|
||||
|
||||
---
|
||||
|
||||
## Multi-user — the plan
|
||||
|
||||
Decided with Ray: **stay on SQLite** (Postgres was considered and dropped -- it is a deployment
|
||||
choice, not a capability one, and nothing here contends for writes). Sign-in is either a local
|
||||
username and password or the Cloudflare Zero Trust that already fronts `ipodderx.sdf1.net`, which
|
||||
puts the authenticated identity in `Cf-Access-Authenticated-User-Email`. Feeds, items and files are **shared**; read state and subscriptions are **per user**.
|
||||
|
||||
The point of sharing: two people subscribed to the same show cost one fetch, one parse, and one file
|
||||
on disk. `enclosures.url` is already globally UNIQUE, so the file half is nearly free.
|
||||
|
||||
- [x] **A. Users, sessions, sign-in.** `users` + `sessions` tables, Argon2 hashing, session cookie,
|
||||
`ipx user add|list|passwd|rm`. A proxy header (`trusted_header` in `[web]`) signs in and
|
||||
optionally creates a user -- honoured only from a `trusted_proxies` address, so a LAN client
|
||||
cannot simply assert it. The existing shared token keeps working and resolves to the admin, so
|
||||
the healthcheck and any scripts survive. Login page for direct access.
|
||||
- [ ] **B. Per-user read state.** `entry_state(user_id, feed_id, guid, read, flagged, position)`;
|
||||
the current columns on `entries` migrate into the first user's rows. Unread counts, filters and
|
||||
playback position all become per user.
|
||||
- [ ] **C. Per-user subscriptions.** `subscriptions(user_id, feed_id)`. config.toml stays the feed
|
||||
catalogue; the UI lists only what you subscribe to. Adding a feed someone else already has costs
|
||||
nothing. A feed nobody subscribes to stops being scanned but keeps its files.
|
||||
- [ ] **D. One file, many users.** Auto-download when *any* subscriber wants it; retention never
|
||||
deletes a file another user has starred or not yet played; deleting a download says so when
|
||||
someone else still has it.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-10 — Items, not episodes
|
||||
|
||||
Half the library is text feeds, so the UI no longer calls everything an episode: counts, the search
|
||||
@@ -1185,3 +1612,37 @@ Gotcha worth keeping: three feature names in the plan were wrong against current
|
||||
`rss/with-syndication` does not exist. `librqbit`'s default features drag in OpenSSL; `rust-tls`
|
||||
is the fix. Whole tree is rustls-only now, no C TLS dependency.
|
||||
Next: step 2 — `config.rs` + `db.rs`. (done)
|
||||
|
||||
---
|
||||
|
||||
## How this was built
|
||||
|
||||
Three planned phases, all complete. The dated entries above are the record of each.
|
||||
|
||||
**The engine (steps 1-8)** — repo skeleton with pinned dependencies; `config.rs` + `db.rs` (TOML
|
||||
config, SQLite schema); `feed.rs` (conditional GET, RSS-then-Atom parsing); `download.rs`
|
||||
(streaming downloads, filters, URL dedupe); `retention.rs` (oldest-first quota and age reaper);
|
||||
`ipc.rs` plus the daemon (Unix-socket JSON lines, TTL scheduler, CLI proxying to a running daemon);
|
||||
`torrent.rs` (librqbit, seeding to a ratio or a time, stall abort); OPML import/export and polish.
|
||||
|
||||
**The web front end (steps 9-13)** — axum serving plain HTML and JS, no WASM toolchain, running
|
||||
*inside* the daemon so it reads SQLite and the event bus directly: config hot-reload and the `[web]`
|
||||
section; browsing feeds and entries with `ammonia`-sanitized show notes; media actions (Range
|
||||
streaming, download on demand, delete, read/starred); feed configuration written back to
|
||||
config.toml; live progress over SSE from the existing broadcast bus.
|
||||
|
||||
**Multi-user (steps A-D)** — accounts, sessions and sign-in (Argon2id, session cookie, a trusted
|
||||
proxy header for Cloudflare Zero Trust); per-user read state; per-user subscriptions; and the rules
|
||||
that follow from one file serving everyone. SQLite stayed: Postgres was considered and dropped, as
|
||||
nothing here contends for writes.
|
||||
|
||||
### The live checks each step had to pass
|
||||
|
||||
1. `ipx add <feed>` + `ipx fetch` → file in `download_dir/<Show>/`, row in `enclosures`.
|
||||
2. `ipx fetch` again → no re-download, feed skipped for TTL.
|
||||
3. `ipx daemon &` + `nc -U $XDG_RUNTIME_DIR/ipx.sock`, send `{"cmd":"fetch"}` → JSON events;
|
||||
a concurrent `ipx fetch` proxies to the daemon instead of downloading in parallel.
|
||||
4. Delete a downloaded file by hand, `ipx fetch` → NOT re-downloaded.
|
||||
5. Torrent enclosure → downloads, moves, stops seeding at the configured ratio/time.
|
||||
6. `ipx reap --dry-run` under quota pressure → oldest-first hit list; real run flips rows to
|
||||
`reaped`.
|
||||
303
docs/sso.md
303
docs/sso.md
@@ -1,9 +1,8 @@
|
||||
# Signing in through Cloudflare Zero Trust or Authentik
|
||||
# Signing in through Cloudflare Access and Authentik
|
||||
|
||||
ipx can take the signed-in identity from whatever sits in front of it, instead of asking for a
|
||||
password itself. Both products below do the same thing in the end: they authenticate the person and
|
||||
pass the result to the origin in a **header**. ipx reads that header, finds (or creates) the
|
||||
matching account, and gets on with it.
|
||||
password itself. The proxy authenticates the person and passes the result to ipx in a **header**;
|
||||
ipx reads it, finds (or creates) the matching account, and gets on with it.
|
||||
|
||||
Read [How this is secured](#how-this-is-secured) before exposing anything. The short version: a
|
||||
header is worth exactly as much as the hop that set it, so ipx only believes one from an address you
|
||||
@@ -11,185 +10,173 @@ list.
|
||||
|
||||
---
|
||||
|
||||
## The ipx side (both setups)
|
||||
## How ipodderx.sdf1.net does it
|
||||
|
||||
Checked end to end on 2026-09-12. An earlier version of this page had never been tried against a
|
||||
real setup and pointed at the wrong address.
|
||||
|
||||
```
|
||||
browser ─► Cloudflare Access, app "ipodderx" ─── sign in ───► Authentik (OpenID Connect)
|
||||
─► tunnel "rays-unraid" (the cloudflared container on Tower)
|
||||
─► http://192.168.1.130:8099 ─► ipx
|
||||
```
|
||||
|
||||
Authentik is not in the request path. It is the identity provider Cloudflare Access asks. Access
|
||||
then adds `Cf-Access-Authenticated-User-Email`, the email address Authentik gave it, to every
|
||||
request it forwards through the tunnel, and ipx signs that person in.
|
||||
|
||||
| Piece | Where | Setting |
|
||||
|---|---|---|
|
||||
| Identity provider | Zero Trust → Settings → Authentication | `Authentik`, OpenID Connect; scopes `openid email profile` |
|
||||
| Access application | Zero Trust → Access → Applications → `ipodderx` | Domain `ipodderx.sdf1.net`; identity providers: Authentik only, with instant auth; session 730h; policy *Require Login* allows a list of email addresses |
|
||||
| Tunnel route | Zero Trust → Networks → Tunnels → `rays-unraid` → Public hostnames | `ipodderx.sdf1.net` → HTTP `192.168.1.130:8099` |
|
||||
| DNS | `sdf1.net` | `ipodderx` CNAME to the tunnel, proxied |
|
||||
| ipx | `/mnt/fast/appdata/ipodderx/config.toml`, `[web]` | below |
|
||||
|
||||
```toml
|
||||
[web]
|
||||
enabled = true
|
||||
bind = "0.0.0.0:8099"
|
||||
token = "…" # keep it: it is the admin, used by the healthcheck
|
||||
|
||||
# The header your proxy sets. Empty (the default) disables this whole path.
|
||||
trusted_header = "Cf-Access-Authenticated-User-Email" # Authentik: "X-authentik-username"
|
||||
|
||||
# Addresses allowed to assert that header -- the proxy, and nothing else.
|
||||
trusted_proxies = ["127.0.0.1", "::1"]
|
||||
|
||||
# Create an account the first time the proxy vouches for a name ipx has not seen.
|
||||
bind = "0.0.0.0:8099"
|
||||
trusted_header = "Cf-Access-Authenticated-User-Email"
|
||||
trusted_proxies = ["127.0.0.1", "::1", "192.168.16.1"]
|
||||
auto_create_users = true
|
||||
|
||||
sign_out_url = "/cdn-cgi/access/logout"
|
||||
session_days = 30
|
||||
```
|
||||
|
||||
Restart the daemon after editing. Accounts made this way have **no password**: they can only ever
|
||||
arrive through the proxy. `ipx user list` marks them `proxy only`.
|
||||
Restart ipx after editing it: `docker compose -f /mnt/fast/arcane/projects/content/compose.yaml
|
||||
restart ipodderx`.
|
||||
|
||||
The first account created is an admin. Every later one is an ordinary user, and an ordinary user
|
||||
cannot change global settings or how often feeds are scanned. Promote someone with:
|
||||
### What was missing
|
||||
|
||||
Cloudflare and Authentik were already right. Three things on the ipx side were not:
|
||||
|
||||
1. **`trusted_header` was empty**, which switches the whole proxy path off. ipx ignored the header
|
||||
and asked for a password.
|
||||
2. **`trusted_proxies` listed only `127.0.0.1`.** The tunnel's requests do not come from there;
|
||||
see the next section.
|
||||
3. **The account had the wrong name.** It was made by hand as `rays`, but the header carries
|
||||
`rays@sdf1.net`. With `auto_create_users` on, the first visit would have made a second, empty
|
||||
account. `ipx user rename rays rays@sdf1.net` fixed that without losing anything.
|
||||
|
||||
### The address to trust, and why it is 192.168.16.1
|
||||
|
||||
`cloudflared` runs in its own container and reaches ipx through the host's published port. Docker
|
||||
(iptables firewall backend) masquerades traffic between its bridge networks, so the tunnel's
|
||||
requests arrive from the **gateway of ipx's own network**, `content_default`:
|
||||
|
||||
```sh
|
||||
ipx user list
|
||||
echo -n 'a good password' | ipx user passwd <name> # optional: also lets them sign in directly
|
||||
docker network inspect content_default -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'
|
||||
```
|
||||
|
||||
Local sign-in at `/login` keeps working alongside all of this, which is how you get in from the LAN
|
||||
when the tunnel is down. A brand new database starts with **admin / ipodderx** — change it.
|
||||
That was measured, not assumed. ipx does not log where a request came from, so the addresses were
|
||||
read from the kernel's connection table inside the container while the site was open. (`/proc/net/tcp`
|
||||
lists them in hex.)
|
||||
|
||||
If the `content` project's network is ever recreated, its gateway can change. Check it again, and
|
||||
update `trusted_proxies` to match.
|
||||
|
||||
### Names
|
||||
|
||||
The username is the email address, lower-cased: `rays@sdf1.net`. To sign in at `/login` with a
|
||||
password from the LAN, use that name too.
|
||||
|
||||
To let someone else in, add their address to the Access policy; they need an Authentik account with
|
||||
that email. With `auto_create_users = true` they get an ipx account on their first visit, as an
|
||||
ordinary user with no feeds. An account made before the proxy can be given the name the proxy will
|
||||
send:
|
||||
|
||||
```sh
|
||||
docker exec iPodderX ipx user rename <old name> <email address>
|
||||
```
|
||||
|
||||
### Signing out
|
||||
|
||||
**Sign out** sends someone the proxy signed in to `sign_out_url`, here Cloudflare's
|
||||
`/cdn-cgi/access/logout`. That ends your Access session for **every** Access application,
|
||||
`code.sdf1.net` included: Cloudflare has no way to end just one, and its sign-out page does not send
|
||||
you anywhere afterwards. The next visit goes back through Authentik, which lets you straight in if
|
||||
you are still signed in there. Signing out of Authentik itself is Authentik's own sign-out.
|
||||
|
||||
ipx never shows its password page to someone the proxy vouches for: `/login` sends them on to their
|
||||
feeds.
|
||||
|
||||
### The tile in Authentik's library
|
||||
|
||||
Authentik's library lists Authentik's own applications, and ipodderx signs in through the one
|
||||
called `Cloudflare Access`, so ipodderx needs a bookmark of its own to show up there. It is
|
||||
Applications → Applications → `ipodderx`: no provider, launch URL `https://ipodderx.sdf1.net`, and
|
||||
the iPodderX icon. Like Outline's, it has no policy bindings, so everyone in Authentik sees the
|
||||
tile. Who actually gets in is still up to the Access policy.
|
||||
|
||||
### Check it
|
||||
|
||||
```sh
|
||||
# From Tower itself: not a trusted address, so the header is ignored.
|
||||
curl -s -H 'Accept: application/json' -H 'Cf-Access-Authenticated-User-Email: rays@sdf1.net' \
|
||||
http://192.168.1.130:8099/api/me # -> sign in
|
||||
|
||||
# From a container on a Docker bridge, as cloudflared is: believed.
|
||||
docker run --rm --network bridge mirror.gcr.io/library/busybox wget -qO- \
|
||||
--header 'Accept: application/json' --header 'Cf-Access-Authenticated-User-Email: rays@sdf1.net' \
|
||||
http://192.168.1.130:8099/api/me # -> {"admin":true,"name":"rays@sdf1.net"}
|
||||
```
|
||||
|
||||
Then open `https://ipodderx.sdf1.net` in a private window. Authentik should ask who you are, and
|
||||
ipx should show `rays@sdf1.net` in the sidebar footer without asking for a password.
|
||||
|
||||
---
|
||||
|
||||
## Cloudflare Zero Trust
|
||||
## The ipx settings
|
||||
|
||||
This is what runs `ipodderx.sdf1.net`: a `cloudflared` tunnel to the origin, with an Access
|
||||
application in front of it. Cloudflare authenticates the visitor and adds
|
||||
`Cf-Access-Authenticated-User-Email` to every request it forwards.
|
||||
|
||||
### 1. The tunnel
|
||||
|
||||
In **Zero Trust → Networks → Tunnels**, either use the existing tunnel or create one, then add a
|
||||
public hostname:
|
||||
|
||||
| Field | Value |
|
||||
| Key | What it does |
|
||||
|---|---|
|
||||
| Subdomain / domain | `ipodderx` / `sdf1.net` |
|
||||
| Type | HTTP |
|
||||
| URL | `localhost:8099` (or the LAN address of the box) |
|
||||
| `trusted_header` | The header the proxy sets. Empty, the default, turns the proxy path off. |
|
||||
| `trusted_proxies` | The addresses allowed to set it. Nothing else is believed. |
|
||||
| `auto_create_users` | Make an account the first time the proxy vouches for a name ipx has not seen. |
|
||||
| `sign_out_url` | Where Sign out sends someone the proxy signed in: the proxy's own sign-out. Empty sends them to the sign-in page, where the proxy signs them straight back in. |
|
||||
| `session_days` | How long a password sign-in lasts without use. |
|
||||
|
||||
Use `localhost` when `cloudflared` runs on the same machine as ipx — that keeps the origin request
|
||||
coming from `127.0.0.1`, which is already in `trusted_proxies`. If `cloudflared` runs elsewhere (its
|
||||
own container, another host), put **its** address in `trusted_proxies` instead, and make sure
|
||||
nothing else can reach port 8099.
|
||||
The first account ever created is an admin. Every later one is an ordinary user, who cannot change
|
||||
global settings, a feed's URL or folder, or how often feeds are scanned: the API refuses those with
|
||||
a `403`, not just the UI. Everything else about a feed is theirs alone; see [users.md](users.md).
|
||||
|
||||
### 2. The Access application
|
||||
|
||||
**Zero Trust → Access → Applications → Add an application → Self-hosted**:
|
||||
|
||||
- Application domain: `ipodderx.sdf1.net`
|
||||
- Session duration: whatever suits; ipx keeps its own 30-day session on top.
|
||||
- Add a policy — *Allow*, with a rule such as `Emails` → your address, or `Emails ending in` →
|
||||
your domain. Anyone this policy admits gets an ipx account when `auto_create_users` is on, so keep
|
||||
the policy as narrow as the people you actually want reading your feeds.
|
||||
|
||||
### 3. Point ipx at the header
|
||||
|
||||
```toml
|
||||
trusted_header = "Cf-Access-Authenticated-User-Email"
|
||||
trusted_proxies = ["127.0.0.1", "::1"]
|
||||
```
|
||||
|
||||
The username becomes the email address, lower-cased (`ray@example.com`). That is what shows in the
|
||||
sidebar and what `ipx user list` prints.
|
||||
|
||||
### 4. Check it
|
||||
|
||||
```sh
|
||||
# From the box itself: no header, no session -> the sign-in page.
|
||||
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' -H 'Accept: text/html' http://127.0.0.1:8099/
|
||||
|
||||
# Pretending to be the tunnel (only works because 127.0.0.1 is trusted):
|
||||
curl -s -H 'Cf-Access-Authenticated-User-Email: you@example.com' http://127.0.0.1:8099/api/me
|
||||
```
|
||||
|
||||
Then load `https://ipodderx.sdf1.net` in a browser: Cloudflare should ask who you are, and ipx
|
||||
should show your address in the sidebar footer without ever asking for a password.
|
||||
Local sign-in at `/login` keeps working alongside the proxy, which is how you get in from the LAN
|
||||
when the tunnel is down. So does the shared `[web] token`, which signs in as the admin and is the
|
||||
way back in if you lock yourself out. A brand new database starts with **admin / ipodderx**;
|
||||
change it.
|
||||
|
||||
---
|
||||
|
||||
## Authentik
|
||||
## Authentik in the request path instead
|
||||
|
||||
Authentik does this with a **Proxy Provider** plus an **outpost**, which sits in the request path and
|
||||
adds `X-authentik-username` (also `X-authentik-email`, `X-authentik-name`, `X-authentik-groups`).
|
||||
Not what ipodderx.sdf1.net uses, and **not verified**. Authentik can also sit in front of ipx
|
||||
itself, with a **Proxy Provider** and an **outpost** that adds `X-authentik-username`:
|
||||
|
||||
### 1. Provider
|
||||
|
||||
**Applications → Providers → Create → Proxy Provider**:
|
||||
|
||||
- Name: `ipx`
|
||||
- Authorization flow: your usual (`default-provider-authorization-implicit-consent`)
|
||||
- Mode: **Forward auth (single application)** if an existing reverse proxy fronts ipx, or
|
||||
**Proxy** to let the outpost talk to ipx directly.
|
||||
- External host: `https://ipodderx.example.net`
|
||||
- Internal host (Proxy mode): `http://<ip of the ipx box>:8099`
|
||||
|
||||
### 2. Application and outpost
|
||||
|
||||
**Applications → Create**, bind it to that provider, and give it a policy so only the people you
|
||||
mean are let through. Then add the provider to an outpost (**Applications → Outposts**, the embedded
|
||||
one is fine).
|
||||
|
||||
### 3. Forward auth, if you use nginx/SWAG in front
|
||||
|
||||
In the server block for ipx:
|
||||
|
||||
```nginx
|
||||
location /outpost.goauthentik.io {
|
||||
proxy_pass http://authentik-server:9000/outpost.goauthentik.io;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
||||
add_header Set-Cookie $auth_cookie;
|
||||
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
||||
}
|
||||
|
||||
location / {
|
||||
auth_request /outpost.goauthentik.io/auth/nginx;
|
||||
error_page 401 = @goauthentik_proxy_signin;
|
||||
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
||||
add_header Set-Cookie $auth_cookie;
|
||||
|
||||
# This is the line that matters to ipx.
|
||||
auth_request_set $authentik_username $upstream_http_x_authentik_username;
|
||||
proxy_set_header X-authentik-username $authentik_username;
|
||||
|
||||
proxy_pass http://ipx:8099;
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Point ipx at the header
|
||||
|
||||
```toml
|
||||
trusted_header = "X-authentik-username"
|
||||
trusted_proxies = ["172.18.0.5"] # the outpost or nginx container, NOT a whole subnet
|
||||
```
|
||||
|
||||
Usernames arrive as Authentik knows them (`ray`), lower-cased.
|
||||
- Applications → Providers → Create → Proxy Provider; mode **Proxy** (the outpost talks to ipx) or
|
||||
**Forward auth** (an existing reverse proxy asks the outpost).
|
||||
- Applications → Create, bound to that provider, with a policy; add the provider to an outpost.
|
||||
- In ipx: `trusted_header = "X-authentik-username"`, and the outpost's or reverse proxy's address
|
||||
in `trusted_proxies`. Measure that address as above rather than guessing it.
|
||||
|
||||
---
|
||||
|
||||
## How this is secured
|
||||
|
||||
**The header is only believed from `trusted_proxies`.** Every other source is ignored, and the
|
||||
request falls through to a session cookie or the shared token. This is the whole security boundary,
|
||||
so:
|
||||
request falls through to a session cookie or the shared token. That is the whole security boundary.
|
||||
|
||||
- List the **proxy's own address**, not a range. `["127.0.0.1"]` when the tunnel runs beside ipx;
|
||||
the container's IP when it does not.
|
||||
- Never list a LAN subnet. Anyone on your network could then send
|
||||
`Cf-Access-Authenticated-User-Email: admin@…` and be your admin.
|
||||
- Make sure the origin port is not reachable *around* the proxy by anyone you would not admit
|
||||
through it. If it is, bind ipx to `127.0.0.1` and let only the proxy reach it.
|
||||
With the tunnel reaching ipx through the host's port, `192.168.16.1` means **any container on Tower
|
||||
that connects to `192.168.1.130:8099`**, not only `cloudflared`. Machines on the LAN, and Tower
|
||||
itself, arrive under their own addresses and cannot set the header; the checks above show both
|
||||
sides. Never list a LAN address or range: anyone there could then send
|
||||
`Cf-Access-Authenticated-User-Email: rays@sdf1.net` and be you.
|
||||
|
||||
Verify the refusal, don't assume it — set `trusted_proxies = ["10.9.9.9"]` briefly and confirm a
|
||||
header from your machine gets a `401`:
|
||||
**What ipx does not do:** it does not verify Cloudflare's signed `Cf-Access-Jwt-Assertion`. It
|
||||
trusts the hop. Verifying the signature would make the containers on Tower irrelevant to the
|
||||
boundary, and is the upgrade if that ever matters.
|
||||
|
||||
```sh
|
||||
curl -s -o /dev/null -w '%{http_code}\n' \
|
||||
-H 'Cf-Access-Authenticated-User-Email: someone@example.com' http://127.0.0.1:8099/api/me
|
||||
```
|
||||
|
||||
**What ipx does not do:** it does not verify Cloudflare's `Cf-Access-Jwt-Assertion` signature or
|
||||
Authentik's session. It trusts the hop. That is a deliberate trade — it keeps the configuration to
|
||||
three lines — and it is sound exactly as long as the point above holds.
|
||||
|
||||
**Turning it off:** clear `trusted_header`. Existing proxy-only accounts stay, but nobody can sign
|
||||
**Turning it off:** clear `trusted_header` and restart. Proxy-made accounts stay, but nobody can sign
|
||||
in with them until they are given a password (`ipx user passwd <name>`).
|
||||
|
||||
---
|
||||
@@ -197,16 +184,20 @@ in with them until they are given a password (`ipx user passwd <name>`).
|
||||
## Everyday administration
|
||||
|
||||
```sh
|
||||
ipx user list # who exists, and how each one signs in
|
||||
ipx user list # who exists, how each signs in, and when
|
||||
echo -n 'secret123' | ipx user add sam # local account, password on stdin
|
||||
ipx user add sam --no-password # proxy-only account, created ahead of time
|
||||
ipx user add sam@example.com --no-password # proxy-only account, made ahead of time
|
||||
ipx user rename sam sam@example.com # give an account the name the proxy sends
|
||||
echo -n 'newsecret' | ipx user passwd sam # change a password
|
||||
ipx user rm sam # remove the account
|
||||
```
|
||||
|
||||
Set `auto_create_users = false` once everyone who should have an account has one. After that the
|
||||
proxy vouching for an unknown name is logged and refused, rather than quietly making an account.
|
||||
In the container, put `docker exec iPodderX` in front, and `docker exec -i iPodderX` for the ones
|
||||
that read a password.
|
||||
|
||||
Scanning intervals, the disk quota, retention and the download folder are **admin-only** — the
|
||||
Settings button is hidden for everyone else, and the API refuses the change even if the request is
|
||||
made by hand. Ordinary users still control their own folders, keywords and downloads per feed.
|
||||
Set `auto_create_users = false` once everyone who should have an account has one. After that the
|
||||
proxy vouching for an unknown name is logged and refused. Make people ahead of time instead, with
|
||||
the exact name the header will carry.
|
||||
|
||||
See also [users.md](users.md) for what several people share, [configuration.md](configuration.md)
|
||||
for every `[web]` key, and [cli.md](cli.md) for the `ipx user` commands.
|
||||
|
||||
98
docs/users.md
Normal file
98
docs/users.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Accounts, and what several people share
|
||||
|
||||
ipx serves any number of people from one copy of the data. The rule that decides everything else:
|
||||
**there is one file on disk per enclosure URL.** Two people subscribed to the same show cost one
|
||||
fetch, one parse and one file.
|
||||
|
||||
## What is yours, what is everyone's
|
||||
|
||||
| Yours alone | The same for everyone |
|
||||
|---|---|
|
||||
| Read, kept, playback position | The feed's URL |
|
||||
| Which feeds you see at all | Its download folder |
|
||||
| Keywords, auto-download, explicit, per-scan cap | When it is scanned |
|
||||
| | The file on disk |
|
||||
|
||||
The right-hand column describes the feed and the file rather than a preference — two people wanting
|
||||
different folders would mean two copies. Those three are **admin-only**, and the API returns `403`
|
||||
for anyone else rather than merely hiding the controls.
|
||||
|
||||
## How the scanner merges everyone's wants
|
||||
|
||||
One fetch serves every subscriber, so the policy is a union:
|
||||
|
||||
* an item is downloaded if **anyone** wants it — one person's keyword set matching is enough, and
|
||||
one person with no keywords removes the filter for that feed entirely
|
||||
* auto-download is on if **anyone** has it on
|
||||
* the per-scan cap is the **largest** anyone asked for
|
||||
|
||||
So "auto-download off" means *I don't cause downloads*, not *I never see them*. If someone else's
|
||||
subscription pulls an item, you see it listed as downloaded and can play it, because the enclosure
|
||||
is shared.
|
||||
|
||||
## Deleting
|
||||
|
||||
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
|
||||
for everyone** and names them in the confirmation, and the server has the last word: if anyone else
|
||||
has kept the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
|
||||
reason, and only `?force=true` goes through.
|
||||
|
||||
Retention follows the same rule: an item anyone kept keeps its file, and it counts as read only once
|
||||
every subscriber has read it.
|
||||
|
||||
## Signing in
|
||||
|
||||
Three ways, tried in order of how specific the claim is:
|
||||
|
||||
1. **A proxy header** naming the user — Cloudflare Zero Trust or Authentik. Honoured only from an
|
||||
address in `trusted_proxies`. See [sso.md](sso.md).
|
||||
2. **A session cookie** from signing in at `/login`. Argon2id hashes, sessions in the database,
|
||||
idle timeout `session_days`.
|
||||
3. **The shared `[web] token`**, which signs in as the admin — this is what the Docker healthcheck
|
||||
and any scripts use.
|
||||
|
||||
A database with no accounts creates **admin / ipodderx** on the next daemon start and says so in
|
||||
the log. Change it:
|
||||
|
||||
```sh
|
||||
echo -n 'a good password' | ipx user passwd admin
|
||||
```
|
||||
|
||||
## Adding someone
|
||||
|
||||
```sh
|
||||
echo -n 'their password' | ipx user add sam
|
||||
```
|
||||
|
||||
They sign in at `/login` and start with **no feeds**: subscriptions are per person. Adding a feed
|
||||
someone else already has costs nothing — no second fetch, no second copy — it just appears on their
|
||||
list with their own read state. Unsubscribing removes it from their list alone; only when the last
|
||||
subscriber leaves does the feed stop being scanned, and even then its files and history stay, so
|
||||
re-subscribing does not pull the back catalogue again.
|
||||
|
||||
**Popular** and **Directory** sit at the top of the feed list, above your own feeds. Popular, also
|
||||
shown in the Add feed dialog, lists the ten feeds with the most subscribers on this server, you
|
||||
included. Directory lists every one of them A to Z. Your own feeds are marked Subscribed.
|
||||
It shows a title, artwork and a count, never a URL or who reads it. An OPML subscription is listed
|
||||
as the feeds inside it, one by one, and never the OPML itself, so you can take just the shows you
|
||||
want. Anything that looks private is left out: a login configured for the feed, credentials in its URL,
|
||||
or a key such as `auth=` or `token=` in the query, or a feed from a paid-feed service such as
|
||||
Patreon or Supercast, which put the key in the path, and any feed inside an OPML that looks private
|
||||
itself. Those are someone's paid subscriptions, and listing them would let anyone here read what
|
||||
they pay for.
|
||||
|
||||
An admin can do the same from **Settings → Manage users…**: add someone (with a password, or none
|
||||
for someone the proxy signs in), tick or untick Admin, or remove an account. Removing one takes its
|
||||
subscriptions and read state with it; downloaded files stay. The only admin cannot be demoted or
|
||||
removed there, so there is always someone who can manage the rest.
|
||||
|
||||
## Admin
|
||||
|
||||
The first account is an admin. An admin can change global settings (scanning interval, quota,
|
||||
retention, media types, download folder), a feed's URL, folder and schedule, and who has an account
|
||||
and who else is an admin, and read the log, which names everyone's feeds and sign-ins. Everyone else
|
||||
gets the Settings and Log buttons hidden and a `403` if they ask anyway.
|
||||
|
||||
```sh
|
||||
ipx user list # the admin column says who
|
||||
```
|
||||
@@ -26,9 +26,6 @@ pub struct General {
|
||||
/// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d".
|
||||
/// A feed's own `schedule` overrides this.
|
||||
pub schedule: String,
|
||||
/// Superseded by `schedule`. Still read so existing configs keep working.
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub interval_mins: Option<u64>,
|
||||
pub organize: Organize,
|
||||
/// 0 = unlimited.
|
||||
pub max_total_gb: f64,
|
||||
@@ -83,6 +80,10 @@ pub struct Web {
|
||||
pub trusted_proxies: Vec<String>,
|
||||
/// Create an account the first time the proxy vouches for a name it has not seen.
|
||||
pub auto_create_users: bool,
|
||||
/// Where Sign out sends someone the proxy signed in. Signing out of ipx alone cannot stick
|
||||
/// while the proxy still vouches for them, so this is the proxy's own sign-out:
|
||||
/// `/cdn-cgi/access/logout` behind Cloudflare Access. Empty sends them to /login.
|
||||
pub sign_out_url: String,
|
||||
/// Sign a session out after this long without a request.
|
||||
pub session_days: i64,
|
||||
}
|
||||
@@ -96,6 +97,7 @@ impl Default for Web {
|
||||
trusted_header: String::new(),
|
||||
trusted_proxies: vec!["127.0.0.1".into(), "::1".into()],
|
||||
auto_create_users: true,
|
||||
sign_out_url: String::new(),
|
||||
session_days: 30,
|
||||
}
|
||||
}
|
||||
@@ -153,7 +155,6 @@ impl Default for General {
|
||||
download_dir: home().join("Podcasts"),
|
||||
socket: default_socket(),
|
||||
schedule: "every 60m".into(),
|
||||
interval_mins: None,
|
||||
organize: Organize::Feed,
|
||||
max_total_gb: 0.0,
|
||||
max_age_days: 0,
|
||||
@@ -176,8 +177,8 @@ impl Default for Torrent {
|
||||
}
|
||||
|
||||
impl General {
|
||||
/// Minutes between checks. Falls back to the legacy `interval_mins`, then to an hour.
|
||||
/// A malformed value warns rather than stopping the daemon.
|
||||
/// Minutes between checks, or an hour when `schedule` is empty or unreadable. A malformed
|
||||
/// value warns rather than stopping the daemon.
|
||||
pub fn interval(&self) -> u64 {
|
||||
if let Some(n) = parse_interval(&self.schedule) {
|
||||
return n;
|
||||
@@ -185,7 +186,7 @@ impl General {
|
||||
if !self.schedule.trim().is_empty() {
|
||||
tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default");
|
||||
}
|
||||
self.interval_mins.filter(|n| *n > 0).unwrap_or(60)
|
||||
60
|
||||
}
|
||||
}
|
||||
|
||||
@@ -278,9 +279,7 @@ pub fn config_path() -> PathBuf {
|
||||
if let Ok(p) = std::env::var("IPX_CONFIG") {
|
||||
return PathBuf::from(p);
|
||||
}
|
||||
dirs::config_dir()
|
||||
.unwrap_or_else(|| home().join(".config"))
|
||||
.join("ipx/config.toml")
|
||||
xdg("XDG_CONFIG_HOME", ".config").join("ipx/config.toml")
|
||||
}
|
||||
|
||||
/// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`.
|
||||
@@ -288,9 +287,7 @@ pub fn data_dir() -> PathBuf {
|
||||
if let Ok(p) = std::env::var("IPX_DATA_DIR") {
|
||||
return PathBuf::from(p);
|
||||
}
|
||||
dirs::data_dir()
|
||||
.unwrap_or_else(|| home().join(".local/share"))
|
||||
.join("ipx")
|
||||
xdg("XDG_DATA_HOME", ".local/share").join("ipx")
|
||||
}
|
||||
|
||||
fn default_socket() -> PathBuf {
|
||||
@@ -346,8 +343,16 @@ pub fn unique_slug(text: &str, taken: &BTreeMap<String, Feed>) -> String {
|
||||
(2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap()
|
||||
}
|
||||
|
||||
/// `$var`, or `~/fallback` when it is unset or empty, as the XDG base directory spec says.
|
||||
fn xdg(var: &str, fallback: &str) -> PathBuf {
|
||||
std::env::var_os(var)
|
||||
.filter(|v| !v.is_empty())
|
||||
.map(PathBuf::from)
|
||||
.unwrap_or_else(|| home().join(fallback))
|
||||
}
|
||||
|
||||
fn home() -> PathBuf {
|
||||
dirs::home_dir().unwrap_or_else(|| PathBuf::from("."))
|
||||
std::env::var_os("HOME").map(PathBuf::from).unwrap_or_else(|| PathBuf::from("."))
|
||||
}
|
||||
|
||||
fn expand_tilde(p: &Path) -> PathBuf {
|
||||
@@ -367,6 +372,8 @@ mod tests {
|
||||
r#"
|
||||
[general]
|
||||
download_dir = "/tmp/pods"
|
||||
# A key older versions read. An old config that still has it has to load.
|
||||
interval_mins = 45
|
||||
|
||||
[feeds.example]
|
||||
url = "https://example.com/feed.xml"
|
||||
@@ -403,22 +410,17 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn interval_falls_back_through_legacy_then_default() {
|
||||
fn interval_falls_back_to_an_hour() {
|
||||
let mut g = General::default();
|
||||
assert_eq!(g.interval(), 60, "the default schedule");
|
||||
|
||||
g.schedule = "every 15m".into();
|
||||
assert_eq!(g.interval(), 15);
|
||||
|
||||
// A config written before `schedule` existed still works.
|
||||
// Empty or garbage must not stop the daemon.
|
||||
g.schedule = String::new();
|
||||
g.interval_mins = Some(45);
|
||||
assert_eq!(g.interval(), 45);
|
||||
|
||||
// Garbage must not stop the daemon.
|
||||
assert_eq!(g.interval(), 60);
|
||||
g.schedule = "whenever".into();
|
||||
assert_eq!(g.interval(), 45);
|
||||
g.interval_mins = None;
|
||||
assert_eq!(g.interval(), 60);
|
||||
}
|
||||
|
||||
|
||||
@@ -223,7 +223,7 @@ enum Sniffed {
|
||||
/// 2008 and so always answered 'data'.
|
||||
async fn sniff(path: &Path) -> Result<Sniffed> {
|
||||
let head = read_head(path, 512).await?;
|
||||
if infer::is(&head, "torrent") || head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
|
||||
if head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
|
||||
return Ok(Sniffed::Torrent);
|
||||
}
|
||||
let text = String::from_utf8_lossy(&head);
|
||||
|
||||
175
src/feed.rs
175
src/feed.rs
@@ -117,6 +117,103 @@ pub fn opml_title(bytes: &[u8]) -> Option<String> {
|
||||
.filter(|t| !t.is_empty())
|
||||
}
|
||||
|
||||
/// The token and show of a Patreon feed link, or None for any other URL.
|
||||
///
|
||||
/// Patreon gives each patron one token per creator. With no show it stands for the creator,
|
||||
/// whose feed carries every show at once.
|
||||
fn patreon_parts(url: &str) -> Option<(String, Option<String>)> {
|
||||
let u = url::Url::parse(url).ok()?;
|
||||
if !matches!(u.host_str()?, "patreon.com" | "www.patreon.com") || !u.path().starts_with("/rss") {
|
||||
return None;
|
||||
}
|
||||
let param = |name: &str| u.query_pairs().find(|(k, _)| k == name).map(|(_, v)| v.into_owned());
|
||||
Some((param("auth")?, param("show")))
|
||||
}
|
||||
|
||||
/// A Patreon link naming a creator but no show.
|
||||
pub fn is_patreon_creator(url: &str) -> bool {
|
||||
matches!(patreon_parts(url), Some((_, None)))
|
||||
}
|
||||
|
||||
/// What was typed into Add feed, as a URL. A bare Patreon token is taken as its creator's
|
||||
/// feed, since the token alone says whose it is.
|
||||
pub fn expand_input(input: &str) -> String {
|
||||
let s = input.trim();
|
||||
let token = s.len() >= 20 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_');
|
||||
if token { format!("https://www.patreon.com/rss?auth={s}") } else { s.to_owned() }
|
||||
}
|
||||
|
||||
/// Whether two URLs are the same feed. One Patreon show has several spellings -- by the
|
||||
/// creator's name, by number, or with no creator at all -- and the token and show are what
|
||||
/// identify it.
|
||||
pub fn same_feed(a: &str, b: &str) -> bool {
|
||||
a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b))
|
||||
}
|
||||
|
||||
/// A Patreon creator's name and shows, each show as (title, feed URL).
|
||||
///
|
||||
/// ponytail: Patreon's own web API, undocumented, asked without signing in. If it changes,
|
||||
/// finding shows stops and the show feeds already found keep working. The documented API
|
||||
/// needs an OAuth client per install and does not list shows.
|
||||
pub async fn patreon_shows(
|
||||
client: &reqwest::Client,
|
||||
url: &str,
|
||||
) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
|
||||
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
|
||||
let mut resp = client.get(url).send().await.context("connecting")?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
|
||||
}
|
||||
let mut head = Vec::new();
|
||||
while patreon_campaign(&head).is_none() && head.len() < 64 * 1024 {
|
||||
let Some(chunk) = resp.chunk().await.context("reading the feed")? else { break };
|
||||
head.extend_from_slice(&chunk);
|
||||
}
|
||||
let campaign = patreon_campaign(&head)
|
||||
.ok_or_else(|| anyhow!("the Patreon feed does not say whose it is"))?;
|
||||
|
||||
let api = format!(
|
||||
"https://www.patreon.com/api/campaigns/{campaign}\
|
||||
?include=shows&fields%5Bcampaign%5D=name&fields%5Bcollection%5D=title"
|
||||
);
|
||||
let resp = client.get(api).send().await.context("asking Patreon for the shows")?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(anyhow!("Patreon would not list the shows: HTTP {}", resp.status()));
|
||||
}
|
||||
let (name, shows) = parse_patreon_shows(&resp.bytes().await.context("reading the shows")?)?;
|
||||
Ok((name, shows.into_iter().map(|(id, title)| (title, format!("{url}&show={id}"))).collect()))
|
||||
}
|
||||
|
||||
/// The campaign number in the start of a Patreon feed.
|
||||
fn patreon_campaign(head: &[u8]) -> Option<String> {
|
||||
let text = String::from_utf8_lossy(head);
|
||||
text.match_indices("patreon.com/rss/").find_map(|(i, m)| {
|
||||
let id: String = text[i + m.len()..].chars().take_while(char::is_ascii_digit).collect();
|
||||
(!id.is_empty()).then_some(id)
|
||||
})
|
||||
}
|
||||
|
||||
/// A campaign's name and its shows as (id, title), from Patreon's JSON:API answer.
|
||||
fn parse_patreon_shows(json: &[u8]) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||
let v: serde_json::Value = serde_json::from_slice(json).context("Patreon's answer is not JSON")?;
|
||||
// Missing is not the same as none. Read as no shows, the creator feed would be scanned as
|
||||
// a plain feed, claim every show's files, and leave the shows empty once the list returned.
|
||||
let ids = v["data"]["relationships"]["shows"]["data"]
|
||||
.as_array()
|
||||
.ok_or_else(|| anyhow!("Patreon's answer does not list the shows"))?;
|
||||
let title = |id: &str| -> Option<String> {
|
||||
let show = v["included"].as_array()?.iter().find(|x| x["type"] == "collection" && x["id"] == id)?;
|
||||
show["attributes"]["title"].as_str().map(|t| t.trim().to_owned())
|
||||
};
|
||||
let shows = ids
|
||||
.iter()
|
||||
.filter_map(|s| s["id"].as_str())
|
||||
.map(|id| (id.to_owned(), title(id).unwrap_or_else(|| format!("Show {id}"))))
|
||||
.collect();
|
||||
Ok((v["data"]["attributes"]["name"].as_str().map(str::to_owned), shows))
|
||||
}
|
||||
|
||||
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
|
||||
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
|
||||
match rss::Channel::read_from(bytes) {
|
||||
@@ -243,7 +340,7 @@ fn from_rss(ch: rss::Channel, bytes: &[u8]) -> ParsedFeed {
|
||||
link: non_empty(item.link()),
|
||||
published: item.pub_date().and_then(parse_date),
|
||||
// Content wins over description, as __getEntries preferred entry.content.
|
||||
description: non_empty(item.content()).or_else(|| non_empty(item.description())),
|
||||
description: body(item.content(), item.description()),
|
||||
categories: item
|
||||
.categories()
|
||||
.iter()
|
||||
@@ -309,11 +406,7 @@ fn from_atom(feed: atom_syndication::Feed) -> ParsedFeed {
|
||||
title: non_empty(Some(e.title().as_str())),
|
||||
link: alt.map(str::to_owned),
|
||||
published: e.published().or(Some(e.updated())).map(|d| d.timestamp()),
|
||||
description: e
|
||||
.content()
|
||||
.and_then(|c| c.value())
|
||||
.or_else(|| e.summary().map(|s| s.as_str()))
|
||||
.map(str::to_owned),
|
||||
description: body(e.content().and_then(|c| c.value()), e.summary().map(|s| s.as_str())),
|
||||
categories: e.categories().iter().map(|c| c.term().to_owned()).collect(),
|
||||
explicit: false,
|
||||
image: None,
|
||||
@@ -358,6 +451,25 @@ fn non_empty(s: Option<&str>) -> Option<String> {
|
||||
s.map(str::trim).filter(|s| !s.is_empty()).map(str::to_owned)
|
||||
}
|
||||
|
||||
/// An item's show notes: its full body when that is whole, else its description.
|
||||
///
|
||||
/// libsyn served Daily Meditation Podcast's `content:encoded` cut at the `>` inside a class name
|
||||
/// pasted from a web app (`[&:has([data-writing-block])>*]:pointer-events-auto`), so the body
|
||||
/// began halfway through a tag and the page showed the rest of the tag as text. The same item's
|
||||
/// `description` was whole. With no description to fall back on, a damaged body beats none.
|
||||
fn body(content: Option<&str>, description: Option<&str>) -> Option<String> {
|
||||
non_empty(content)
|
||||
.filter(|c| !starts_mid_tag(c))
|
||||
.or_else(|| non_empty(description))
|
||||
.or_else(|| non_empty(content))
|
||||
}
|
||||
|
||||
/// Text that closes an attribute list (`">`) before any tag has opened is the tail of a tag whose
|
||||
/// start was cut off.
|
||||
fn starts_mid_tag(html: &str) -> bool {
|
||||
html[..html.find('<').unwrap_or(html.len())].contains("\">")
|
||||
}
|
||||
|
||||
/// The picture to show beside an item, in order of how deliberate it is:
|
||||
/// `itunes:image`, then Media RSS `media:thumbnail`, then a `media:content` that is an
|
||||
/// image, and finally an image enclosure -- which is how a blog's article picture arrives
|
||||
@@ -451,6 +563,18 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_body_cut_off_mid_tag_gives_way_to_the_description() {
|
||||
// How libsyn served Daily Meditation Podcast #3477: content:encoded began inside a tag.
|
||||
let cut = r#"*]:pointer-events-auto R6Vx5W_threadScrollVars" dir="auto" data-turn="assistant"> <p>What if</p>"#;
|
||||
let whole = r#"<div class="[&:has([data-writing-block])>*]:pointer-events-auto"><p>What if</p></div>"#;
|
||||
assert_eq!(body(Some(cut), Some(whole)).as_deref(), Some(whole));
|
||||
assert_eq!(body(Some("<p>Notes</p>"), Some("Summary")).as_deref(), Some("<p>Notes</p>"), "a whole body wins");
|
||||
assert_eq!(body(Some("Plain notes, no tags."), Some("Summary")).as_deref(), Some("Plain notes, no tags."));
|
||||
assert_eq!(body(Some(cut), None).as_deref(), Some(cut), "a damaged body beats none");
|
||||
assert_eq!(body(None, Some("Summary")).as_deref(), Some("Summary"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn feed_level_explicit_overrides_entries() {
|
||||
let xml = br#"<?xml version="1.0"?>
|
||||
@@ -550,6 +674,45 @@ mod tests {
|
||||
assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_patreon_creator_is_a_list_of_its_shows() {
|
||||
let tok = "AbCdEfGhIjKlMnOpQrStUvWxYz012_-9";
|
||||
assert_eq!(expand_input(&format!(" {tok} ")), format!("https://www.patreon.com/rss?auth={tok}"));
|
||||
assert_eq!(expand_input("https://example.com/rss"), "https://example.com/rss");
|
||||
|
||||
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss/glasscannon?auth={tok}")));
|
||||
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss?auth={tok}")));
|
||||
assert!(!is_patreon_creator(&format!("https://www.patreon.com/rss/x?auth={tok}&show=1")), "one show is a feed");
|
||||
assert!(!is_patreon_creator(&format!("https://example.com/rss?auth={tok}")));
|
||||
|
||||
// The show you already have by name is the one a bare token would add by number.
|
||||
assert!(same_feed(
|
||||
&format!("https://www.patreon.com/rss/glasscannon?auth={tok}&show=2073588"),
|
||||
&format!("https://www.patreon.com/rss?auth={tok}&show=2073588"),
|
||||
));
|
||||
assert!(!same_feed(
|
||||
&format!("https://www.patreon.com/rss?auth={tok}&show=1"),
|
||||
&format!("https://www.patreon.com/rss?auth={tok}&show=2"),
|
||||
));
|
||||
|
||||
// The self link carries the campaign by number, whichever spelling was asked for.
|
||||
let head = br#"<rss><channel><link>https://www.patreon.com/glasscannon</link>
|
||||
<atom:link href="https://www.patreon.com/rss/369921?auth=t" rel="self"/>"#;
|
||||
assert_eq!(patreon_campaign(head).as_deref(), Some("369921"));
|
||||
assert_eq!(patreon_campaign(b"<rss><channel><title>T"), None);
|
||||
|
||||
let json = br#"{"data":{"id":"369921","type":"campaign","attributes":{"name":"The Glass Cannon Network"},
|
||||
"relationships":{"shows":{"data":[{"id":"2073588","type":"collection"},{"id":"2073636","type":"collection"}]}}},
|
||||
"included":[{"id":"2073588","type":"collection","attributes":{"title":"Get in the Trunk "}},
|
||||
{"id":"2073636","type":"collection","attributes":{"title":"Shadowdark"}}]}"#;
|
||||
let (name, shows) = parse_patreon_shows(json).unwrap();
|
||||
assert_eq!(name.as_deref(), Some("The Glass Cannon Network"));
|
||||
assert_eq!(shows, [("2073588".into(), "Get in the Trunk".into()), ("2073636".into(), "Shadowdark".into())]);
|
||||
|
||||
// An answer that stops naming the shows is an error, never "this creator has none".
|
||||
assert!(parse_patreon_shows(br#"{"data":{"attributes":{"name":"X"}}}"#).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_item_may_carry_several_enclosures() {
|
||||
// The rss crate keeps only one per item -- the last -- so these come from the XML.
|
||||
|
||||
61
src/ipc.rs
61
src/ipc.rs
@@ -62,10 +62,10 @@ impl Event {
|
||||
Event::DownloadDone { path, .. } => format!(" saved {path}"),
|
||||
Event::DownloadError { url, msg, .. } => format!(" failed {url}: {msg}"),
|
||||
Event::Reaped { path, bytes } => {
|
||||
format!("reap {path} ({:.1} MB)", *bytes as f64 / 1_048_576.0)
|
||||
format!("deleted {path} ({:.1} MB)", *bytes as f64 / 1_048_576.0)
|
||||
}
|
||||
Event::ReapDone { files, bytes } => format!(
|
||||
"reaped {files} file(s), {:.1} MB",
|
||||
"deleted {files} old file(s), {:.1} MB",
|
||||
*bytes as f64 / 1_048_576.0
|
||||
),
|
||||
Event::Status { feeds, pending, downloaded } => {
|
||||
@@ -172,11 +172,18 @@ pub async fn daemon_is_live(path: &Path) -> bool {
|
||||
UnixStream::connect(path).await.is_ok()
|
||||
}
|
||||
|
||||
/// Answers `status` for the socket, without the worker. The worker runs one job at a time, and a
|
||||
/// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon
|
||||
/// dead. The answer goes to the client that asked and no one else: broadcast, it ended any
|
||||
/// `ipx fetch` that was watching a scan, since `status` is a terminal event.
|
||||
pub type StatusFn = std::sync::Arc<dyn Fn() -> Event + Send + Sync>;
|
||||
|
||||
/// Accepts connections, feeding commands to `cmds` and events from `events` back out.
|
||||
pub async fn serve(
|
||||
path: PathBuf,
|
||||
events: broadcast::Sender<Event>,
|
||||
cmds: mpsc::Sender<Command>,
|
||||
status: StatusFn,
|
||||
) -> Result<()> {
|
||||
// A socket file left by a crashed daemon would block the bind; a live one was already
|
||||
// rejected by the caller's daemon_is_live() check.
|
||||
@@ -195,8 +202,9 @@ pub async fn serve(
|
||||
let (stream, _) = listener.accept().await?;
|
||||
let rx = events.subscribe();
|
||||
let cmds = cmds.clone();
|
||||
let status = status.clone();
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = handle(stream, rx, cmds).await {
|
||||
if let Err(e) = handle(stream, rx, cmds, status).await {
|
||||
tracing::debug!(error = %e, "client gone");
|
||||
}
|
||||
});
|
||||
@@ -207,12 +215,21 @@ async fn handle(
|
||||
stream: UnixStream,
|
||||
mut rx: broadcast::Receiver<Event>,
|
||||
cmds: mpsc::Sender<Command>,
|
||||
status: StatusFn,
|
||||
) -> Result<()> {
|
||||
let (read, mut write) = stream.into_split();
|
||||
|
||||
// Events out.
|
||||
// Events out: everything broadcast, and the answers meant for this client alone.
|
||||
let (reply, mut replies) = mpsc::channel::<Event>(4);
|
||||
let writer = tokio::spawn(async move {
|
||||
while let Ok(ev) = rx.recv().await {
|
||||
loop {
|
||||
let ev = tokio::select! {
|
||||
Some(ev) = replies.recv() => ev,
|
||||
got = rx.recv() => match got {
|
||||
Ok(ev) => ev,
|
||||
Err(_) => break,
|
||||
},
|
||||
};
|
||||
let mut line = serde_json::to_string(&ev).unwrap_or_default();
|
||||
line.push('\n');
|
||||
if write.write_all(line.as_bytes()).await.is_err() {
|
||||
@@ -229,6 +246,15 @@ async fn handle(
|
||||
continue;
|
||||
}
|
||||
match serde_json::from_str::<Command>(line) {
|
||||
// Answered here, not queued behind whatever the worker is on: see StatusFn.
|
||||
Ok(Command::Status) => {
|
||||
tracing::info!(target: "ipx::io", "-> {line}");
|
||||
let ev = status();
|
||||
if let Ok(json) = serde_json::to_string(&ev) {
|
||||
tracing::info!(target: "ipx::io", "<- {json}");
|
||||
}
|
||||
let _ = reply.send(ev).await;
|
||||
}
|
||||
Ok(cmd) => {
|
||||
if cmds.send(cmd).await.is_err() {
|
||||
break; // Worker is gone; so are we.
|
||||
@@ -329,4 +355,29 @@ mod tests {
|
||||
}
|
||||
.is_terminal());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn status_is_answered_while_the_worker_is_busy() {
|
||||
// The queue is full and nobody drains it, as when the worker is deep in a long download:
|
||||
// anything sent to it would wait for ever.
|
||||
let (cmds, _worker) = mpsc::channel::<Command>(1);
|
||||
cmds.send(Command::Reap { dry_run: true }).await.unwrap();
|
||||
let (events, _) = broadcast::channel::<Event>(8);
|
||||
// Another client, watching a scan: it must not be handed someone else's answer, which
|
||||
// would end its session.
|
||||
let mut watcher = events.subscribe();
|
||||
let status: StatusFn = std::sync::Arc::new(|| Event::Status { feeds: 1, pending: 2, downloaded: 3 });
|
||||
let (client, server) = UnixStream::pair().unwrap();
|
||||
tokio::spawn(handle(server, events.subscribe(), cmds, status));
|
||||
|
||||
let (read, mut write) = client.into_split();
|
||||
write.write_all(b"{\"cmd\":\"status\"}\n").await.unwrap();
|
||||
let line = tokio::time::timeout(std::time::Duration::from_secs(2), BufReader::new(read).lines().next_line())
|
||||
.await
|
||||
.expect("status waited behind the worker")
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert!(line.contains(r#""ev":"status""#), "{line}");
|
||||
assert!(watcher.try_recv().is_err(), "the answer went to every client, not just the one asking");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -111,18 +111,11 @@ impl Visit for Collect {
|
||||
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
||||
self.add(field, format!("{value:?}"));
|
||||
}
|
||||
// Numbers and bools reach record_debug through the trait's defaults, which prints them the
|
||||
// same way. A string would print quoted there, hence its own method.
|
||||
fn record_str(&mut self, field: &Field, value: &str) {
|
||||
self.add(field, value.to_owned());
|
||||
}
|
||||
fn record_i64(&mut self, field: &Field, value: i64) {
|
||||
self.add(field, value.to_string());
|
||||
}
|
||||
fn record_u64(&mut self, field: &Field, value: u64) {
|
||||
self.add(field, value.to_string());
|
||||
}
|
||||
fn record_bool(&mut self, field: &Field, value: bool) {
|
||||
self.add(field, value.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
486
src/main.rs
486
src/main.rs
@@ -99,6 +99,9 @@ enum UserCmd {
|
||||
Passwd { name: String },
|
||||
/// Delete an account and everything it knows: its subscriptions and read state
|
||||
Rm { name: String },
|
||||
/// Rename an account, keeping its feeds, read state and admin rights. This is how an
|
||||
/// account made before the proxy takes the name the proxy signs it in as
|
||||
Rename { name: String, new_name: String },
|
||||
}
|
||||
|
||||
/// What a brand new database starts with, so there is always a way in. Announced loudly
|
||||
@@ -273,11 +276,16 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
||||
println!("no accounts yet: ipx user add <name>");
|
||||
}
|
||||
for u in users {
|
||||
let added = u
|
||||
.created
|
||||
.and_then(|t| chrono::DateTime::from_timestamp(t, 0))
|
||||
.map_or("?".into(), |d| d.format("%Y-%m-%d").to_string());
|
||||
let seen = u.last_login.map_or("never signed in".into(), |t| format!("signed in {}", ago(Some(t))));
|
||||
println!(
|
||||
"{:<20} {:<8} {}",
|
||||
"{:<20} {:<6} {:<11} added {added} {seen}",
|
||||
u.name,
|
||||
if u.is_admin { "admin" } else { "" },
|
||||
if u.pass_hash.is_some() { "password" } else { "proxy only" }
|
||||
if u.pass_hash.is_some() { "password" } else { "proxy only" },
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
@@ -292,6 +300,22 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
||||
println!("password changed for {name}");
|
||||
Ok(())
|
||||
}
|
||||
UserCmd::Rename { name, new_name } => {
|
||||
let name = name.trim().to_ascii_lowercase();
|
||||
// The same rules as a name the proxy vouches for, or the proxy would never find it.
|
||||
let new_name = crate::auth::name_from_header(&new_name)
|
||||
.ok_or_else(|| anyhow::anyhow!("not a usable name: no commas, semicolons or line breaks"))?;
|
||||
let user = ctx
|
||||
.db
|
||||
.user_by_name(&name)?
|
||||
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
|
||||
if ctx.db.user_by_name(&new_name)?.is_some() {
|
||||
anyhow::bail!("{new_name} already exists");
|
||||
}
|
||||
ctx.db.rename_user(user.id, &new_name)?;
|
||||
println!("renamed {name} to {new_name}");
|
||||
Ok(())
|
||||
}
|
||||
UserCmd::Rm { name } => {
|
||||
let name = name.trim().to_ascii_lowercase();
|
||||
let user = ctx
|
||||
@@ -315,14 +339,24 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
|
||||
Cmd::Reap { dry_run } => reap(ctx, dry_run, true),
|
||||
Cmd::Download { enclosure } => download_one(ctx, enclosure).await,
|
||||
Cmd::Status => {
|
||||
let (pending, downloaded) = ctx.db.counts()?;
|
||||
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0);
|
||||
ctx.out.emit(Event::Status { feeds, pending, downloaded });
|
||||
ctx.out.emit(status(ctx));
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The counts `ipx status` prints. A running daemon's socket answers with this directly rather
|
||||
/// than through the job queue.
|
||||
fn status(ctx: &Ctx) -> Event {
|
||||
match ctx.db.counts() {
|
||||
Ok((pending, downloaded)) => {
|
||||
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0);
|
||||
Event::Status { feeds, pending, downloaded }
|
||||
}
|
||||
Err(e) => Event::Error { msg: format!("{e:#}") },
|
||||
}
|
||||
}
|
||||
|
||||
async fn daemon(
|
||||
ctx: Arc<Ctx>,
|
||||
config_path: PathBuf,
|
||||
@@ -334,8 +368,7 @@ async fn daemon(
|
||||
anyhow::bail!("a daemon is already listening on {}", socket.display());
|
||||
}
|
||||
|
||||
// A database with nobody in it cannot be signed into, and an install that predates
|
||||
// accounts still has to serve its owner. Both get the same starting point.
|
||||
// A database with nobody in it cannot be signed into.
|
||||
if ctx.db.users()?.is_empty() {
|
||||
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
|
||||
tracing::warn!(
|
||||
@@ -344,10 +377,13 @@ async fn daemon(
|
||||
);
|
||||
}
|
||||
|
||||
match migrate_opml_children(&ctx) {
|
||||
Ok(n) if n > 0 => tracing::info!(count = n, "moved OPML feeds out of config.toml into the database"),
|
||||
Ok(_) => {}
|
||||
Err(e) => tracing::warn!(error = ?e, "could not tidy OPML feeds out of the config"),
|
||||
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
|
||||
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
|
||||
match ctx.db.adopt_catalogue(admin.id, &catalogue) {
|
||||
Ok(0) => {}
|
||||
Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
|
||||
Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
|
||||
}
|
||||
}
|
||||
|
||||
match ctx.db.requeue_interrupted() {
|
||||
@@ -359,7 +395,12 @@ async fn daemon(
|
||||
let (tx_cmd, mut rx_cmd) = mpsc::channel::<Cmd>(64);
|
||||
|
||||
let web = start_web(&ctx, &config_path, web_addr, &tx_cmd, &events).await?;
|
||||
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd));
|
||||
// status is answered by the socket itself; everything else waits its turn in the queue.
|
||||
let answer: ipc::StatusFn = {
|
||||
let ctx = ctx.clone();
|
||||
Arc::new(move || status(&ctx))
|
||||
};
|
||||
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer));
|
||||
|
||||
// One command at a time: the queue is what keeps two scans from overlapping.
|
||||
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
|
||||
@@ -458,7 +499,7 @@ async fn start_web(
|
||||
let mut fresh = (*cfg).clone();
|
||||
fresh.web.enabled = true;
|
||||
fresh.web.bind = bind.clone();
|
||||
fresh.web.token = web::generate_token();
|
||||
fresh.web.token = crate::auth::new_session_token();
|
||||
fresh.save(config_path)?;
|
||||
ctx.reload_cfg(config_path)?;
|
||||
println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token);
|
||||
@@ -507,8 +548,9 @@ async fn add(
|
||||
keywords: Vec<String>,
|
||||
) -> Result<()> {
|
||||
let mut cfg = (*ctx.cfg()).clone();
|
||||
let url = &feed::expand_input(url);
|
||||
// Includes feeds derived from an OPML, or the same show could be added twice.
|
||||
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| s.cfg.url == url) {
|
||||
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
|
||||
anyhow::bail!("already subscribed as {:?}", existing.id);
|
||||
}
|
||||
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
|
||||
@@ -559,10 +601,17 @@ pub async fn add_one(
|
||||
|
||||
// Slugs must be unique across derived feeds too, or a new feed can collide with one
|
||||
// an OPML already introduced.
|
||||
let taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
|
||||
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
|
||||
.into_iter()
|
||||
.map(|s| (s.id, s.cfg))
|
||||
.collect();
|
||||
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
|
||||
// it gets its history back, and a different feed does not inherit someone else's.
|
||||
for (id, other) in ctx.db.feed_urls()? {
|
||||
if !feed::same_feed(&other, url) {
|
||||
taken.entry(id).or_insert_with(|| probe.clone());
|
||||
}
|
||||
}
|
||||
let id = config::unique_slug(&title, &taken);
|
||||
cfg.feeds.insert(id.clone(), probe);
|
||||
Ok(id)
|
||||
@@ -592,44 +641,94 @@ fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> {
|
||||
}
|
||||
|
||||
async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path) -> Result<()> {
|
||||
let mut cfg = (*ctx.cfg()).clone();
|
||||
let text = std::fs::read_to_string(file)
|
||||
.with_context(|| format!("reading {}", file.display()))?;
|
||||
let doc = opml::OPML::from_str(&text).map_err(|e| anyhow::anyhow!("parsing OPML: {e}"))?;
|
||||
// The CLI speaks for the operator, as the shared web token does.
|
||||
let admin = ctx
|
||||
.db
|
||||
.users()?
|
||||
.into_iter()
|
||||
.find(|u| u.is_admin)
|
||||
.ok_or_else(|| anyhow::anyhow!("no admin account to subscribe: ipx user add <name> --admin"))?;
|
||||
let doc = opml::OPML::from_str(&text)
|
||||
.map_err(|e| anyhow::anyhow!("{} is not OPML: {e}", file.display()))?;
|
||||
let (added, had) = subscribe_opml(ctx, config_path, &doc, admin.id)?;
|
||||
println!("subscribed {} to {added} feed(s); {had} already there", admin.name);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Subscribes one person to every feed in an OPML document, for the CLI and the web alike.
|
||||
/// A feed already in the catalogue costs nothing; an unknown one is added under the OPML's
|
||||
/// title rather than refetching each. Returns (newly subscribed, already subscribed).
|
||||
///
|
||||
/// Before accounts, importing only added unknown URLs to config.toml. Once subscriptions
|
||||
/// decided what each person sees, that imported nothing at all for a feed someone else
|
||||
/// already had, and a new one had no subscriber, so it was never scanned.
|
||||
///
|
||||
/// The caller parses the document, so each refuses a file that is not OPML in its own terms,
|
||||
/// before anything is touched: a 400 from the web, a message from the CLI.
|
||||
pub fn subscribe_opml(
|
||||
ctx: &Ctx,
|
||||
config_path: &std::path::Path,
|
||||
doc: &opml::OPML,
|
||||
user_id: i64,
|
||||
) -> Result<(usize, usize)> {
|
||||
let mut found = vec![];
|
||||
collect_outlines(&doc.body.outlines, &mut found);
|
||||
|
||||
let mut added = 0;
|
||||
let known = subscriptions(ctx)?;
|
||||
let mut cfg = (*ctx.cfg()).clone();
|
||||
let mut ids = vec![];
|
||||
let mut grew = false;
|
||||
for (title, url) in found {
|
||||
if cfg.feeds.values().any(|f| f.url == url) {
|
||||
continue;
|
||||
}
|
||||
// Name it from the OPML title rather than refetching every feed.
|
||||
let id = config::unique_slug(&title, &cfg.feeds);
|
||||
cfg.feeds.insert(
|
||||
id.clone(),
|
||||
config::Feed {
|
||||
url,
|
||||
folder: None,
|
||||
group: None,
|
||||
media_types: None,
|
||||
schedule: None,
|
||||
keywords: vec![],
|
||||
allow_explicit: false,
|
||||
auto_download: true,
|
||||
max_new_per_check: None,
|
||||
username: None,
|
||||
password: None,
|
||||
password_env: None,
|
||||
},
|
||||
);
|
||||
println!("added {id}");
|
||||
added += 1;
|
||||
let existing = known
|
||||
.iter()
|
||||
.find(|s| s.cfg.url == url)
|
||||
.map(|s| s.id.clone())
|
||||
// The same URL listed twice in one file.
|
||||
.or_else(|| cfg.feeds.iter().find(|(_, f)| f.url == url).map(|(id, _)| id.clone()));
|
||||
let id = match existing {
|
||||
Some(id) => id,
|
||||
None => {
|
||||
let id = config::unique_slug(&title, &cfg.feeds);
|
||||
cfg.feeds.insert(
|
||||
id.clone(),
|
||||
config::Feed {
|
||||
url,
|
||||
folder: None,
|
||||
group: None,
|
||||
media_types: None,
|
||||
schedule: None,
|
||||
keywords: vec![],
|
||||
allow_explicit: false,
|
||||
auto_download: true,
|
||||
max_new_per_check: None,
|
||||
username: None,
|
||||
password: None,
|
||||
password_env: None,
|
||||
},
|
||||
);
|
||||
grew = true;
|
||||
id
|
||||
}
|
||||
};
|
||||
ids.push(id);
|
||||
}
|
||||
cfg.save(config_path)?;
|
||||
println!("{added} feed(s) imported");
|
||||
Ok(())
|
||||
if grew {
|
||||
cfg.save(config_path)?;
|
||||
ctx.reload_cfg(config_path)?;
|
||||
}
|
||||
|
||||
let (mut added, mut had) = (0, 0);
|
||||
for id in ids {
|
||||
if ctx.db.subscription(user_id, &id)?.is_some() {
|
||||
had += 1;
|
||||
} else {
|
||||
ctx.db.subscribe(user_id, &id)?;
|
||||
added += 1;
|
||||
}
|
||||
}
|
||||
Ok((added, had))
|
||||
}
|
||||
|
||||
/// OPML nests feeds inside folder outlines, so this walks the whole tree.
|
||||
@@ -747,7 +846,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
|
||||
ctx.out.emit(Event::FeedSkip {
|
||||
feed: id.clone(),
|
||||
reason: format!(
|
||||
"OPML: {total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
|
||||
"{total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
|
||||
added.len()
|
||||
),
|
||||
});
|
||||
@@ -848,36 +947,6 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Moves OPML children that older versions wrote into config.toml over to the database.
|
||||
/// They were never yours to edit, and 80-odd of them made the file unreadable.
|
||||
fn migrate_opml_children(ctx: &Ctx) -> Result<usize> {
|
||||
let cfg = (*ctx.cfg()).clone();
|
||||
let children: Vec<(String, config::Feed)> = cfg
|
||||
.feeds
|
||||
.iter()
|
||||
.filter(|(_, f)| f.group.is_some())
|
||||
.map(|(id, f)| (id.clone(), f.clone()))
|
||||
.collect();
|
||||
if children.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let mut fresh = cfg.clone();
|
||||
for (id, f) in &children {
|
||||
let group = f.group.clone().unwrap_or_default();
|
||||
let title = ctx
|
||||
.db
|
||||
.feed_summary(id)
|
||||
.ok()
|
||||
.and_then(|s| s.title)
|
||||
.unwrap_or_else(|| id.clone());
|
||||
ctx.db.upsert_managed(id, &f.url, &title, &group)?;
|
||||
fresh.feeds.remove(id);
|
||||
}
|
||||
fresh.save(&ctx.config_path)?;
|
||||
ctx.reload_cfg(&ctx.config_path)?;
|
||||
Ok(children.len())
|
||||
}
|
||||
|
||||
/// Seconds to wait before re-checking a feed.
|
||||
///
|
||||
/// A per-feed schedule is an explicit instruction and wins outright. Without one, the
|
||||
@@ -903,7 +972,7 @@ struct Scan {
|
||||
enum Outcome {
|
||||
NotModified,
|
||||
Feed(Scan),
|
||||
/// The URL served an OPML document, so it is a subscription list rather than a feed.
|
||||
/// The URL is a list of feeds rather than a feed: an OPML, or a Patreon creator's shows.
|
||||
Opml { added: Vec<String>, removed: usize, kept: usize, total: usize },
|
||||
}
|
||||
|
||||
@@ -913,6 +982,31 @@ async fn scan_one(
|
||||
feed_cfg: &config::Feed,
|
||||
state: &db::HttpState,
|
||||
) -> Result<Outcome> {
|
||||
// A Patreon creator with more than one show is a list of feeds, like an OPML.
|
||||
if feed::is_patreon_creator(&feed_cfg.url) {
|
||||
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
|
||||
Ok((name, shows)) if shows.len() > 1 => {
|
||||
ctx.db.touch_feed(id, &feed_cfg.url)?;
|
||||
if let Some(name) = name {
|
||||
ctx.db.set_title(id, &name)?;
|
||||
}
|
||||
// Read as one feed before it was split, it listed every show's items in one
|
||||
// heap. The items go; its files and read state move to each show as the show
|
||||
// lists them (`Db::adopt`), so no show comes up empty for want of a URL.
|
||||
ctx.db.clear_entries(id)?;
|
||||
return sync_group(ctx, id, feed_cfg, &shows).await;
|
||||
}
|
||||
Ok(_) => {} // One show: the creator's feed is that show.
|
||||
// Already split: keep the shows it has rather than read the creator as one heap.
|
||||
Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e),
|
||||
Err(e) => tracing::warn!(
|
||||
feed = id,
|
||||
error = %format!("{e:#}"),
|
||||
"could not list the Patreon shows; reading it as one feed"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
let mut fetched = feed::fetch(
|
||||
&ctx.client,
|
||||
feed_cfg,
|
||||
@@ -957,29 +1051,45 @@ async fn scan_one(
|
||||
parsed.image.as_deref(),
|
||||
)?;
|
||||
|
||||
let policy = policy_for(ctx, id, feed_cfg)?;
|
||||
if let Some(parent) = &feed_cfg.group {
|
||||
let listed: Vec<(&str, &str)> = parsed
|
||||
.entries
|
||||
.iter()
|
||||
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
|
||||
.collect();
|
||||
ctx.db.adopt(parent, id, &listed)?;
|
||||
}
|
||||
// Verdicts are recorded in `state`, so the download queue below is just "everything still
|
||||
// pending". A filter's verdict is looked at again on every scan, though: made once, at
|
||||
// discovery, it outlived the setting behind it, and allowing explicit items afterwards
|
||||
// changed nothing however often the feed was scanned.
|
||||
let skipped = ctx.db.skipped_by_filter(id)?;
|
||||
let mut scan = Scan::default();
|
||||
for entry in &parsed.entries {
|
||||
if ctx.db.record_entry(id, entry)? {
|
||||
scan.new_entries += 1;
|
||||
}
|
||||
for enc in &entry.enclosures {
|
||||
if !ctx.db.record_enclosure(id, &entry.guid, enc)? {
|
||||
continue; // Seen before: downloaded, skipped or deliberately reaped.
|
||||
}
|
||||
// Filters run once, at discovery, and are recorded in `state`. The download
|
||||
// queue below is then just "everything still pending".
|
||||
if let Some(reason) = reject(&ctx.cfg(), feed_cfg, entry, enc) {
|
||||
ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?;
|
||||
let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? {
|
||||
None
|
||||
} else if let Some(reason) = skipped.get(&enc.url) {
|
||||
Some(reason.as_str())
|
||||
} else {
|
||||
continue; // Settled: queued, downloaded, reaped, or another feed's file.
|
||||
};
|
||||
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
|
||||
if now != was {
|
||||
match now {
|
||||
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?,
|
||||
None => ctx.db.mark_enclosure(&enc.url, "pending", None)?,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An unset per-feed cap follows the global one; 0 there means unlimited.
|
||||
let budget = feed_cfg.max_new_per_check.unwrap_or_else(|| {
|
||||
let g = ctx.cfg().general.max_new_per_check;
|
||||
if g == 0 { usize::MAX } else { g }
|
||||
});
|
||||
if feed_cfg.auto_download && budget > 0 {
|
||||
let budget = policy.budget;
|
||||
if policy.auto_download && budget > 0 {
|
||||
let cfg = ctx.cfg();
|
||||
let folder = download::folder_for(&cfg, id, feed_cfg, parsed.title.as_deref());
|
||||
let dest_dir = cfg.general.download_dir.join(&folder);
|
||||
@@ -1057,12 +1167,6 @@ async fn scan_one(
|
||||
Ok(Outcome::Feed(scan))
|
||||
}
|
||||
|
||||
/// Brings the feed list in step with a subscribed OPML.
|
||||
///
|
||||
/// New entries are added under the OPML's group and folder. An entry that has gone from
|
||||
/// the OPML is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
|
||||
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
|
||||
/// the UI to explain them.
|
||||
async fn sync_opml(
|
||||
ctx: &Arc<Ctx>,
|
||||
parent_id: &str,
|
||||
@@ -1073,25 +1177,44 @@ async fn sync_opml(
|
||||
if let Some(title) = feed::opml_title(bytes) {
|
||||
ctx.db.set_title(parent_id, &title)?;
|
||||
}
|
||||
sync_group(ctx, parent_id, parent, &listed).await
|
||||
}
|
||||
|
||||
/// Brings the feed list in step with a list of feeds: a subscribed OPML, or a Patreon
|
||||
/// creator's shows.
|
||||
///
|
||||
/// New entries are added under the list's group and folder. An entry that has gone from
|
||||
/// the list is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
|
||||
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
|
||||
/// the UI to explain them.
|
||||
async fn sync_group(
|
||||
ctx: &Arc<Ctx>,
|
||||
parent_id: &str,
|
||||
parent: &config::Feed,
|
||||
listed: &[(String, String)],
|
||||
) -> Result<Outcome> {
|
||||
let cfg = ctx.cfg();
|
||||
let existing = ctx.db.managed_feeds()?;
|
||||
let mut added = vec![];
|
||||
|
||||
for (title, url) in &listed {
|
||||
for (title, url) in listed {
|
||||
// Already known, whether derived or promoted into the config.
|
||||
if let Some(m) = existing.iter().find(|m| &m.url == url) {
|
||||
ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
|
||||
continue;
|
||||
}
|
||||
if cfg.feeds.values().any(|f| &f.url == url) {
|
||||
// A Patreon show you added by hand may be spelled differently from the one listed.
|
||||
if cfg.feeds.values().any(|f| feed::same_feed(&f.url, url)) {
|
||||
continue;
|
||||
}
|
||||
// A removed feed keeps its rows, so its id is only free again for the same feed.
|
||||
let known = ctx.db.feed_urls()?;
|
||||
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
|
||||
.feeds
|
||||
.keys()
|
||||
.chain(existing.iter().map(|m| &m.id))
|
||||
.chain(added.iter())
|
||||
.chain(known.iter().filter(|(_, u)| !feed::same_feed(u, url)).map(|(id, _)| id))
|
||||
.map(|id| (id.clone(), parent.clone()))
|
||||
.collect();
|
||||
let id = config::unique_slug(title, &taken);
|
||||
@@ -1099,6 +1222,23 @@ async fn sync_opml(
|
||||
added.push(id);
|
||||
}
|
||||
|
||||
// Whoever subscribes to the OPML subscribes to what it lists: that is what taking a
|
||||
// subscription means. Their own feeds are untouched.
|
||||
for id in ctx
|
||||
.db
|
||||
.managed_feeds()?
|
||||
.iter()
|
||||
.filter(|m| m.group_id == parent_id)
|
||||
.map(|m| m.id.clone())
|
||||
.chain(std::iter::once(parent_id.to_string()))
|
||||
{
|
||||
for user in ctx.db.users()? {
|
||||
if ctx.db.subscription(user.id, parent_id)?.is_some() {
|
||||
ctx.db.subscribe(user.id, &id)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Anything in this group the OPML no longer lists.
|
||||
let mut removed = 0;
|
||||
let mut kept = 0;
|
||||
@@ -1125,11 +1265,12 @@ async fn sync_opml(
|
||||
fn reject(
|
||||
cfg: &config::Config,
|
||||
feed_cfg: &config::Feed,
|
||||
policy: &Policy,
|
||||
entry: &feed::Entry,
|
||||
enc: &feed::Enclosure,
|
||||
) -> Option<&'static str> {
|
||||
let url = enc.url.as_str();
|
||||
if !feed_cfg.auto_download {
|
||||
if !policy.auto_download {
|
||||
return Some("auto_download is off");
|
||||
}
|
||||
// Blog feeds put the article's header image in an <enclosure>; without this a text
|
||||
@@ -1141,7 +1282,7 @@ fn reject(
|
||||
if !config::wanted_media(enc.mime.as_deref(), wanted) {
|
||||
return Some("not audio or video");
|
||||
}
|
||||
if entry.explicit && !feed_cfg.allow_explicit {
|
||||
if entry.explicit && !policy.allow_explicit {
|
||||
return Some("explicit");
|
||||
}
|
||||
let categories = entry.categories.join(" ");
|
||||
@@ -1151,12 +1292,79 @@ fn reject(
|
||||
entry.description.as_deref().unwrap_or(""),
|
||||
categories.as_str(),
|
||||
];
|
||||
if !download::matches_keywords(&feed_cfg.keywords, &haystacks) {
|
||||
// One file serves everyone subscribed, so an item is wanted if it is wanted by
|
||||
// anyone: any one person's keyword set matching is enough.
|
||||
let wanted_by_someone = policy.keyword_sets.is_empty()
|
||||
|| policy
|
||||
.keyword_sets
|
||||
.iter()
|
||||
.any(|set| download::matches_keywords(set, &haystacks));
|
||||
if !wanted_by_someone {
|
||||
return Some("no keyword match");
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// What the scanner should do for a feed, merged across everyone subscribed to it. The
|
||||
/// feed is fetched once and its files are downloaded once, so the merge is a union: if
|
||||
/// one person wants a thing, it is fetched, and everyone else simply sees it listed.
|
||||
///
|
||||
/// With no subscribers at all -- a hand-written config entry nobody has claimed yet --
|
||||
/// the feed's own settings stand, which is how a single-user install behaves.
|
||||
pub struct Policy {
|
||||
pub auto_download: bool,
|
||||
pub allow_explicit: bool,
|
||||
/// Empty means take everything. Otherwise one set per subscriber who filters.
|
||||
pub keyword_sets: Vec<Vec<String>>,
|
||||
pub budget: usize,
|
||||
}
|
||||
|
||||
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
|
||||
let global = ctx.cfg().general.max_new_per_check;
|
||||
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global))
|
||||
}
|
||||
|
||||
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
|
||||
let cap = |n: Option<usize>| n.unwrap_or(if global == 0 { usize::MAX } else { global });
|
||||
|
||||
if subs.is_empty() {
|
||||
return Policy {
|
||||
auto_download: feed_cfg.auto_download,
|
||||
allow_explicit: feed_cfg.allow_explicit,
|
||||
keyword_sets: if feed_cfg.keywords.is_empty() {
|
||||
vec![]
|
||||
} else {
|
||||
vec![feed_cfg.keywords.clone()]
|
||||
},
|
||||
budget: cap(feed_cfg.max_new_per_check),
|
||||
};
|
||||
}
|
||||
|
||||
let mut policy = Policy {
|
||||
auto_download: false,
|
||||
allow_explicit: false,
|
||||
keyword_sets: vec![],
|
||||
budget: 0,
|
||||
};
|
||||
for sub in subs {
|
||||
if !sub.auto_download.unwrap_or(feed_cfg.auto_download) {
|
||||
continue; // Not fetching for this person, so their wants add nothing.
|
||||
}
|
||||
policy.auto_download = true;
|
||||
policy.allow_explicit |= sub.allow_explicit.unwrap_or(feed_cfg.allow_explicit);
|
||||
policy.budget = policy
|
||||
.budget
|
||||
.max(cap(sub.max_new_per_check.map(|n| n as usize).or(feed_cfg.max_new_per_check)));
|
||||
let kw = sub.keywords.clone().unwrap_or_else(|| feed_cfg.keywords.clone());
|
||||
if kw.is_empty() {
|
||||
// Somebody takes everything, so no filter can apply to the shared copy.
|
||||
return Policy { keyword_sets: vec![], ..policy };
|
||||
}
|
||||
policy.keyword_sets.push(kw);
|
||||
}
|
||||
policy
|
||||
}
|
||||
|
||||
async fn fetch_one(
|
||||
ctx: &Arc<Ctx>,
|
||||
feed_id: &str,
|
||||
@@ -1352,3 +1560,79 @@ fn duration(secs: u64) -> String {
|
||||
s => format!("{}d", s / 86_400),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn feed() -> config::Feed {
|
||||
// Whatever `ipx add` would write, which is the shape every code path sees.
|
||||
let mut cfg = config::Config::default();
|
||||
let f = add_one_cfg(&mut cfg, "http://x/f.xml", None, vec![]);
|
||||
f
|
||||
}
|
||||
|
||||
/// The feed entry `add` builds, without the network round trip it does for a title.
|
||||
fn add_one_cfg(
|
||||
_cfg: &mut config::Config,
|
||||
url: &str,
|
||||
folder: Option<String>,
|
||||
keywords: Vec<String>,
|
||||
) -> config::Feed {
|
||||
config::Feed {
|
||||
url: url.into(),
|
||||
folder,
|
||||
keywords,
|
||||
allow_explicit: false,
|
||||
auto_download: true,
|
||||
group: None,
|
||||
media_types: None,
|
||||
schedule: None,
|
||||
max_new_per_check: None,
|
||||
username: None,
|
||||
password: None,
|
||||
password_env: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn sub(kw: Option<&[&str]>, auto: Option<bool>, max: Option<i64>) -> db::Sub {
|
||||
db::Sub {
|
||||
feed_id: "f".into(),
|
||||
keywords: kw.map(|k| k.iter().map(|s| s.to_string()).collect()),
|
||||
auto_download: auto,
|
||||
allow_explicit: None,
|
||||
max_new_per_check: max,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_shared_feed_is_fetched_for_whoever_wants_the_most() {
|
||||
// Nobody subscribed: the feed's own settings stand, as in a single-user install.
|
||||
let p = merge_policy(&[], &feed(), 3);
|
||||
assert!(p.auto_download);
|
||||
assert_eq!(p.budget, 3);
|
||||
assert!(p.keyword_sets.is_empty());
|
||||
|
||||
// Two filters: an item wanted by either of them is fetched, since one file serves
|
||||
// both. The larger per-scan cap wins for the same reason.
|
||||
let p = merge_policy(
|
||||
&[sub(Some(&["rust"]), None, Some(2)), sub(Some(&["sqlite"]), None, Some(9))],
|
||||
&feed(),
|
||||
3,
|
||||
);
|
||||
assert_eq!(p.keyword_sets.len(), 2);
|
||||
assert_eq!(p.budget, 9);
|
||||
|
||||
// One person taking everything removes the filter for the shared copy.
|
||||
let p = merge_policy(&[sub(Some(&["rust"]), None, None), sub(Some(&[]), None, None)], &feed(), 3);
|
||||
assert!(p.keyword_sets.is_empty());
|
||||
|
||||
// Everyone has auto-download off: nothing is fetched automatically.
|
||||
let p = merge_policy(&[sub(None, Some(false), None), sub(None, Some(false), None)], &feed(), 3);
|
||||
assert!(!p.auto_download);
|
||||
|
||||
// One of them wants it, so it is fetched.
|
||||
let p = merge_policy(&[sub(None, Some(false), None), sub(None, Some(true), None)], &feed(), 3);
|
||||
assert!(p.auto_download);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -150,33 +150,52 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn query_never_offers_flagged_files_and_prefers_read_ones() {
|
||||
fn query_never_offers_a_file_anyone_starred_and_prefers_ones_everyone_read() {
|
||||
// One file serves both subscribers, so it takes both of them to release it.
|
||||
let db = Db::memory().unwrap();
|
||||
db.exec_for_test(
|
||||
"INSERT INTO entries (feed_id, guid, first_seen, read, flagged) VALUES
|
||||
('f', 'keep', 0, 1, 1),
|
||||
('f', 'unread', 0, 0, 0),
|
||||
('f', 'read', 0, 1, 0);
|
||||
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
|
||||
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f');
|
||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
||||
('f', 'keep', 0),
|
||||
('f', 'half', 0),
|
||||
('f', 'unread', 0),
|
||||
('f', 'read', 0);
|
||||
-- Starred by one of the two, so it stays whatever the other thinks.
|
||||
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
|
||||
(1, 'f', 'keep', 1, 1),
|
||||
(2, 'f', 'keep', 1, 0),
|
||||
(1, 'f', 'half', 1, 0),
|
||||
(1, 'f', 'read', 1, 0),
|
||||
(2, 'f', 'read', 1, 0);
|
||||
INSERT INTO enclosures (id, feed_id, guid, url, path, bytes_done, state, downloaded_at) VALUES
|
||||
(1, 'f', 'keep', 'u1', '/tmp/keep', 10, 'done', 10),
|
||||
(2, 'f', 'unread', 'u2', '/tmp/unread', 10, 'done', 20),
|
||||
(3, 'f', 'read', 'u3', '/tmp/read', 10, 'done', 30);",
|
||||
(2, 'f', 'half', 'u2', '/tmp/half', 10, 'done', 20),
|
||||
(3, 'f', 'unread', 'u3', '/tmp/unread', 10, 'done', 30),
|
||||
(4, 'f', 'read', 'u4', '/tmp/read', 10, 'done', 40);",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let got: Vec<i64> = db.reap_candidates().unwrap().iter().map(|c| c.id).collect();
|
||||
assert_eq!(got, vec![3, 2], "flagged excluded; read goes before unread");
|
||||
assert_eq!(
|
||||
got,
|
||||
vec![4, 2, 3],
|
||||
"starred by anyone is never offered; read by everyone goes first, and one \
|
||||
person still having it unread keeps it back with the unread ones"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prune_keeps_entries_that_still_have_a_file() {
|
||||
let db = Db::memory().unwrap();
|
||||
db.exec_for_test(
|
||||
"INSERT INTO entries (feed_id, guid, first_seen, read, flagged) VALUES
|
||||
('f', 'has-file', 100, 1, 0),
|
||||
('f', 'no-file', 100, 1, 0),
|
||||
('f', 'flagged', 100, 1, 1),
|
||||
('f', 'recent', 900, 1, 0);
|
||||
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
|
||||
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',1);
|
||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
||||
('f', 'has-file', 100),
|
||||
('f', 'no-file', 100),
|
||||
('f', 'flagged', 100),
|
||||
('f', 'recent', 900);
|
||||
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
|
||||
(1, 'f', 'has-file', 'u1', '/tmp/x', 'done');",
|
||||
)
|
||||
|
||||
757
src/web.rs
757
src/web.rs
File diff suppressed because it is too large
Load Diff
@@ -15,7 +15,7 @@ const script = html.split('<script>')[1].split('</script>')[0];
|
||||
const ids = new Set([...html.matchAll(/id="([^"]+)"/g)].map(m => m[1]));
|
||||
|
||||
const missing = [];
|
||||
const el = (name) => new Proxy({ style: {}, dataset: {}, classList: { add(){}, remove(){}, toggle(){}, contains(){ return false; } },
|
||||
const el = (name) => new Proxy({ style: { setProperty(){}, getPropertyValue(){ return ''; } }, dataset: {}, classList: { add(){}, remove(){}, toggle(){}, contains(){ return false; } },
|
||||
value: '', textContent: '', innerHTML: '', hidden: false, children: [], firstElementChild: null,
|
||||
appendChild(){}, removeChild(){}, remove(){}, insertAdjacentHTML(){}, addEventListener(){},
|
||||
setAttribute(){}, getAttribute(){ return null; }, select(){}, setSelectionRange(){}, focus(){},
|
||||
@@ -44,7 +44,12 @@ const ctx = {
|
||||
json: () => Promise.resolve(
|
||||
String(url).includes('/api/settings')
|
||||
? { schedule: 'every 60m', every_mins: 60, download_dir: '/tmp', max_total_gb: 0, max_age_days: 0 }
|
||||
: []),
|
||||
: String(url).includes('/api/users')
|
||||
? [{ id: 1, name: 'admin', admin: true, password: true }, { id: 2, name: 'sam', admin: false, password: false }]
|
||||
: /\/api\/(popular|directory)/.test(String(url))
|
||||
? [{ id: 'f', title: 'A Feed', image: null, subscribers: 2, subscribed: true },
|
||||
{ id: 'g', title: null, image: null, subscribers: 1, subscribed: false }]
|
||||
: /entries/.test(String(url)) ? { total: 0, entries: [] } : []),
|
||||
}),
|
||||
EventSource: function () { this.close = () => {}; },
|
||||
MediaMetadata: function () {},
|
||||
@@ -79,6 +84,11 @@ const drive = [
|
||||
['downloadLatestModal', () => ctx.downloadLatestModal(feed)],
|
||||
['removeFeed', () => ctx.removeFeed(feed)],
|
||||
['prefsModal', () => ctx.prefsModal()],
|
||||
['usersModal', () => ctx.usersModal()],
|
||||
['opmlModal', () => ctx.opmlModal()],
|
||||
['selectFeed (directory)', () => ctx.selectFeed(':directory')],
|
||||
['selectFeed (popular)', () => ctx.selectFeed(':popular')],
|
||||
['selectFeed (all subscriptions)', () => ctx.selectFeed(':all')],
|
||||
['logsModal', () => ctx.logsModal()],
|
||||
// `const S` is not reachable from here: top-level const/let do not become properties
|
||||
// of a vm context the way var and function declarations do.
|
||||
|
||||
@@ -10,8 +10,8 @@ test.beforeEach(async ({ page }) => {
|
||||
test('the page loads and lists the configured feeds', async ({ page }) => {
|
||||
// Regression: a ReferenceError in the script left the shell rendered and the sidebar
|
||||
// empty, with every handler below the error dead. Server-side checks all passed.
|
||||
// Three top-level feeds in the fixture config; the OPML's child is inside a closed folder.
|
||||
await expect(page.locator('.feed')).toHaveCount(4, { timeout: 15_000 });
|
||||
// Four top-level feeds in the fixture config; the OPML's children are inside a closed folder.
|
||||
await expect(page.locator('.feed')).toHaveCount(5, { timeout: 15_000 });
|
||||
await expect(page.getByText('Test Show')).toBeVisible();
|
||||
const errors = [];
|
||||
page.on('pageerror', e => errors.push(e.message));
|
||||
@@ -29,6 +29,18 @@ test('the theme toggle actually changes the theme', async ({ page }) => {
|
||||
.not.toBe(before);
|
||||
});
|
||||
|
||||
test('the theme button steps through dark, light and classic, and remembers', async ({ page }) => {
|
||||
const theme = () => page.evaluate(() => document.documentElement.dataset.theme);
|
||||
for (let i = 0; i < 3 && (await theme()) !== 'classic'; i++) await page.locator('#theme').click();
|
||||
expect(await theme()).toBe('classic');
|
||||
await expect(page.locator('#theme')).toHaveAttribute('title', /Classic.*Click for Dark/);
|
||||
|
||||
await page.reload();
|
||||
await expect.poll(theme).toBe('classic');
|
||||
// The 2004 Mac app set its type in Lucida Grande.
|
||||
expect(await page.evaluate(() => getComputedStyle(document.body).fontFamily)).toContain('Lucida Grande');
|
||||
});
|
||||
|
||||
test('settings opens and saves the global schedule', async ({ page }) => {
|
||||
await page.locator('#prefs').click();
|
||||
await expect(page.locator('#modal.on')).toBeVisible();
|
||||
@@ -69,20 +81,21 @@ test('the three panes are there and the item text lands in the bottom one', asyn
|
||||
|
||||
await page.locator('.ep', { hasText: 'First Episode' }).click();
|
||||
await expect(page.locator('#detail .dt')).toHaveText('First Episode');
|
||||
// The enclosure travels with the item, into the same pane.
|
||||
await expect(page.locator('#detail .encbox')).toHaveCount(1);
|
||||
// The enclosure goes to the Files pane beside the list, as the original's did.
|
||||
await expect(page.locator('#files')).toBeVisible();
|
||||
await expect(page.locator('#files .encbox')).toHaveCount(1);
|
||||
|
||||
// Only the downloaded one gets a player, and max_new_per_check is 1, so find it by
|
||||
// its chip rather than assuming which episode the daemon happened to fetch.
|
||||
const downloaded = page.locator('.ep', { hasText: 'downloaded' }).first();
|
||||
const downloaded = page.locator('.ep', { has: page.locator('.kind.here') }).first();
|
||||
await downloaded.click();
|
||||
await expect(page.locator('#detail audio')).toBeVisible();
|
||||
await expect(page.locator('#detail .encbox .btn', { hasText: 'Save' })).toBeVisible();
|
||||
await expect(page.locator('#files [data-a="play"]')).toBeVisible();
|
||||
await expect(page.locator('#files .encbox [title="Save to this computer"]')).toBeVisible();
|
||||
|
||||
// Selecting another item replaces the pane rather than stacking.
|
||||
await page.locator('.ep', { hasText: 'First Episode' }).click();
|
||||
await expect(page.locator('#detail .dt')).toHaveText('First Episode');
|
||||
await expect(page.locator('#detail audio')).toHaveCount(0);
|
||||
await expect(page.locator('#files [data-a="play"]')).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('a downloaded file that is not audio gets no player', async ({ page }) => {
|
||||
@@ -96,12 +109,12 @@ test('a downloaded file that is not audio gets no player', async ({ page }) => {
|
||||
await row.click();
|
||||
|
||||
await expect(page.locator('#detail .dt')).toHaveText('An Article');
|
||||
await expect(page.locator('#detail audio')).toHaveCount(0);
|
||||
await expect(page.locator('#detail .encbox')).toContainText('image');
|
||||
await expect(page.locator('#detail .encbox')).toContainText('downloaded');
|
||||
await expect(page.locator('#files [data-a="play"]')).toHaveCount(0);
|
||||
// What it is and that it is here: one icon, green, with the words in its tooltip.
|
||||
await expect(page.locator('#files .encbox .kind.here')).toHaveAttribute('title', 'image, downloaded');
|
||||
// Still offered as a file, just not as an episode: viewable and keepable.
|
||||
await expect(page.locator('#detail .btn', { hasText: 'Save' })).toBeVisible();
|
||||
const view = page.locator('#detail a', { hasText: 'View' });
|
||||
await expect(page.locator('#files [title="Save to this computer"]')).toBeVisible();
|
||||
const view = page.locator('#files a[title="View in a new tab"]');
|
||||
await expect(view).toHaveAttribute('target', '_blank');
|
||||
await expect(view).toHaveAttribute('rel', /noopener/);
|
||||
await expect(view).toHaveAttribute('href', /\/media\/\d+/);
|
||||
@@ -115,9 +128,9 @@ test('an item with several enclosures lists them all', async ({ page }) => {
|
||||
await expect(row).toContainText('+1 more file');
|
||||
|
||||
await row.click();
|
||||
// The pane below lists every one: the audio and the image.
|
||||
await expect(page.locator('#detail .encbox')).toHaveCount(2);
|
||||
await expect(page.locator('#detail .encbox').nth(1)).toContainText('image');
|
||||
// The Files pane lists every one: the audio and the image.
|
||||
await expect(page.locator('#files .encbox')).toHaveCount(2);
|
||||
await expect(page.locator('#files .encbox').nth(1).locator('.kind[title^="image"]')).toBeVisible();
|
||||
});
|
||||
|
||||
test('the filter tabs change what is listed', async ({ page }) => {
|
||||
@@ -129,13 +142,13 @@ test('the filter tabs change what is listed', async ({ page }) => {
|
||||
await page.locator('.tabs button', { hasText: 'Unread' }).first().click();
|
||||
expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all);
|
||||
|
||||
await page.locator('.tabs button', { hasText: 'Flagged' }).first().click();
|
||||
await page.locator('.tabs button', { hasText: 'Kept' }).first().click();
|
||||
await expect(page.locator('#count')).toContainText('0 items');
|
||||
});
|
||||
|
||||
test('a feed URL is editable and has a copy button', async ({ page }) => {
|
||||
await page.getByText('Test Show').click();
|
||||
await page.locator('.btn', { hasText: 'Settings' }).first().click();
|
||||
await page.locator('#content .acts [data-a="settings"]').click();
|
||||
await expect(page.locator('#surl')).toHaveValue(/show\.xml/);
|
||||
await expect(page.locator('#scopy')).toBeVisible();
|
||||
|
||||
@@ -174,20 +187,56 @@ test('an OPML subscription is a collapsible folder', async ({ page }) => {
|
||||
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
||||
}));
|
||||
|
||||
// Every row reserves the chevron slot for alignment; only a folder's is clickable.
|
||||
// Only a folder has a triangle, and it is a button that says whether the folder is open.
|
||||
const chev = page.locator('.feed.group .chev');
|
||||
await expect(chev).toBeVisible({ timeout: 20_000 });
|
||||
await expect(page.locator('.feed:not(.group) .chev')).toHaveCount(0);
|
||||
await expect(chev).toHaveAttribute('aria-expanded', 'false');
|
||||
|
||||
// Closed by default: the child is not listed until the folder is opened.
|
||||
// Closed by default: the children are not listed until the folder is opened.
|
||||
const before = await page.locator('.feed').count();
|
||||
await chev.click();
|
||||
await expect(page.locator('.feed')).toHaveCount(before + 1);
|
||||
await expect(page.locator('.feed')).toHaveCount(before + 2);
|
||||
// Scoped to the sidebar: the name also appears as the page heading once selected.
|
||||
await expect(page.locator('#feedlist').getByText('Grouped Show')).toBeVisible();
|
||||
|
||||
// The subscription's own page lists what is inside it.
|
||||
await page.locator('.feed', { hasText: 'Test Subscriptions' }).first().click();
|
||||
await expect(page.locator('.childrow')).toHaveCount(1);
|
||||
await expect(page.locator('.childrow')).toHaveCount(2);
|
||||
});
|
||||
|
||||
test('inside an OPML, feeds with unread items are listed first', async ({ page }) => {
|
||||
await page.evaluate(() =>
|
||||
fetch('/api/fetch', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
||||
}));
|
||||
const chev = page.locator('.feed.group .chev');
|
||||
await expect(chev).toBeVisible({ timeout: 20_000 });
|
||||
await chev.click();
|
||||
const side = page.locator('#feedlist');
|
||||
await expect(side.getByText('Aardvark Radio')).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
// Other tests change read state, so set it here: Aardvark Radio read, Grouped Show not.
|
||||
// Opening an item reads it; the toggle in the pane below flips it back.
|
||||
await side.getByText('Aardvark Radio').click();
|
||||
const aa = page.locator('.ep', { hasText: 'Aardvark Ep' });
|
||||
await aa.click();
|
||||
await expect(aa).toHaveClass(/read/);
|
||||
await side.getByText('Grouped Show').click();
|
||||
const gs = page.locator('.ep', { hasText: 'Grouped Ep' });
|
||||
await gs.click();
|
||||
await page.locator('#detail button[title="Mark unread"]').click();
|
||||
await expect(gs).not.toHaveClass(/read/);
|
||||
|
||||
// Aardvark comes first alphabetically and in the OPML, so only the unread sort puts
|
||||
// Grouped Show above it. The folder stays open across the reload (localStorage).
|
||||
await page.reload();
|
||||
const want = ['Grouped Show', 'Aardvark Radio'];
|
||||
await expect(page.locator('#feedlist .feed.child b')).toHaveText(want, { timeout: 20_000 });
|
||||
await page.locator('.feed', { hasText: 'Test Subscriptions' }).first().click();
|
||||
await expect(page.locator('.childrow b')).toHaveText(want);
|
||||
});
|
||||
|
||||
test('marking an OPML subscription read covers the feeds inside it', async ({ page }) => {
|
||||
@@ -204,7 +253,7 @@ test('marking an OPML subscription read covers the feeds inside it', async ({ pa
|
||||
await expect(folder.locator('.badge')).not.toHaveText('0');
|
||||
await folder.click();
|
||||
|
||||
await page.locator('#content .acts button', { hasText: 'Mark all read' }).click();
|
||||
await page.locator('#content .acts [data-a="read"]').click();
|
||||
await expect(folder.locator('.badge')).toHaveText('0');
|
||||
});
|
||||
|
||||
@@ -246,9 +295,9 @@ test('opening an item marks it read, and the toggle flips it back', async ({ pag
|
||||
// Another test may have opened this item already, so start from a known state: the
|
||||
// toggle in the text below flips it back -- which used to recurse until the stack blew.
|
||||
await row().click();
|
||||
await page.locator('#detail button', { hasText: 'Mark unread' }).click();
|
||||
await page.locator('#detail button[title="Mark unread"]').click();
|
||||
await expect(row()).not.toHaveClass(/read/);
|
||||
await expect(page.locator('#detail button', { hasText: 'Mark read' })).toBeVisible();
|
||||
await expect(page.locator('#detail button[title="Mark read"]')).toBeVisible();
|
||||
|
||||
// Opening it is reading it.
|
||||
await row().click();
|
||||
@@ -256,3 +305,508 @@ test('opening an item marks it read, and the toggle flips it back', async ({ pag
|
||||
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
test('the toolbar acts on the selected item', async ({ page }) => {
|
||||
await page.getByText('Test Show').click();
|
||||
const row = () => page.locator('.ep', { hasText: 'Second Episode' });
|
||||
await expect(row()).toBeVisible({ timeout: 20_000 });
|
||||
// Nothing selected, nothing to act on.
|
||||
await expect(page.locator('#tbRead')).toBeDisabled();
|
||||
|
||||
await row().click(); // opening it reads it
|
||||
await expect(row()).toHaveClass(/read/);
|
||||
await page.locator('#tbRead').click();
|
||||
await expect(row()).not.toHaveClass(/read/);
|
||||
|
||||
await page.locator('#tbFlag').click();
|
||||
await expect(row().locator('.fl')).toHaveClass(/on/);
|
||||
await page.locator('#tbFlag').click(); // and back, so later tests see it unkept
|
||||
await expect(row().locator('.fl')).not.toHaveClass(/on/);
|
||||
|
||||
// Second Episode is the one the daemon downloaded, so it plays from the toolbar.
|
||||
await expect(page.locator('#tbPlay')).toBeEnabled();
|
||||
await page.locator('#tbPlay').click();
|
||||
await expect(page.locator('#player')).toBeVisible();
|
||||
await page.locator('#pclose').click();
|
||||
});
|
||||
|
||||
test('a second person has their own feeds and their own read state', async ({ browser }) => {
|
||||
const { execFileSync } = require('child_process');
|
||||
const setup = require('./global-setup');
|
||||
const env = {
|
||||
...process.env,
|
||||
IPX_CONFIG: `${setup.root}/config/config.toml`,
|
||||
IPX_DATA_DIR: `${setup.root}/data`,
|
||||
};
|
||||
try {
|
||||
execFileSync('./target/debug/ipx', ['user', 'add', 'sam'], { input: 'sampassword', env });
|
||||
} catch (e) {
|
||||
if (!String(e.stderr || e.stdout).includes('already exists')) throw e;
|
||||
}
|
||||
|
||||
// A fresh context, so none of the admin's cookies come along.
|
||||
const ctx = await browser.newContext();
|
||||
const page = await ctx.newPage();
|
||||
await page.goto('/login');
|
||||
// The sign-in page shows the icon, so it has to load before anyone has signed in.
|
||||
const icon = await page.request.get('/icon.png');
|
||||
expect(icon.status()).toBe(200);
|
||||
expect(icon.headers()['content-type']).toBe('image/png');
|
||||
await page.locator('#name').fill('sam');
|
||||
await page.locator('#pw').fill('sampassword');
|
||||
await page.locator('button[type=submit]').click();
|
||||
await expect(page.locator('#feedlist')).toBeVisible();
|
||||
|
||||
// Sam subscribes to nothing yet, so sees nothing -- the admin's feeds are not theirs.
|
||||
await expect(page.locator('#feedlist')).toContainText('No feeds.');
|
||||
await expect(page.locator('#prefs')).toBeHidden(); // not an admin
|
||||
// Hiding the button is not the guard; the server is.
|
||||
expect((await page.request.get('/api/users')).status()).toBe(403);
|
||||
await expect(page.locator('#logs')).toBeHidden();
|
||||
expect((await page.request.get('/api/logs')).status()).toBe(403);
|
||||
|
||||
// Subscribing to a feed the admin already has costs no second fetch: same feed, same
|
||||
// files, but Sam's own read state.
|
||||
await page.locator('#addFeed').click();
|
||||
await page.locator('#nurl').fill('http://127.0.0.1:8792/show.xml');
|
||||
await page.locator('#nsave').click();
|
||||
await expect(page.locator('.feed', { hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
await page.locator('.feed', { hasText: 'Test Show' }).click();
|
||||
await expect(page.locator('.ep').first()).toBeVisible({ timeout: 20_000 });
|
||||
// The admin read these earlier in this file; for Sam they are all still unread.
|
||||
const rows = await page.locator('.ep').count();
|
||||
await page.locator('.tabs button', { hasText: 'Unread' }).click();
|
||||
await expect(page.locator('.ep')).toHaveCount(rows);
|
||||
|
||||
// Two people now share this feed, so the page says so and Delete is honest about it.
|
||||
await expect(page.locator('#content .sub').first()).toContainText('shared with 1 other person');
|
||||
|
||||
await ctx.close();
|
||||
});
|
||||
|
||||
test('deleting a shared file warns that it is everyone\'s copy', async ({ page }) => {
|
||||
// Admin and Sam both subscribe to Test Show by now, and the daemon downloaded a file.
|
||||
await page.getByText('Test Show').click();
|
||||
await page.locator('.tabs button', { hasText: 'Downloaded' }).click();
|
||||
const row = page.locator('.ep').first();
|
||||
await expect(row).toBeVisible({ timeout: 20_000 });
|
||||
await row.click();
|
||||
|
||||
// An icon now; what it does, and to whom, is in its tooltip.
|
||||
const del = page.locator('#files button[data-a="del"]');
|
||||
await expect(del).toHaveAttribute('title', /^Delete for everyone \(shared with 1 other person/);
|
||||
|
||||
// Two prompts: the page's own, then the server's, because someone else has not played
|
||||
// it. Accept the first, decline the second, and the file must survive.
|
||||
const seen = [];
|
||||
page.on('dialog', d => {
|
||||
seen.push(d.message());
|
||||
if (seen.length === 1) d.accept();
|
||||
else d.dismiss();
|
||||
});
|
||||
await del.click();
|
||||
await expect.poll(() => seen.length, { timeout: 10_000 }).toBe(2);
|
||||
expect(seen[0]).toContain('shared with 1 other person');
|
||||
expect(seen[1]).toContain('one copy of this file');
|
||||
|
||||
await page.reload();
|
||||
await page.getByText('Test Show').click();
|
||||
await page.locator('.tabs button', { hasText: 'Downloaded' }).click();
|
||||
await expect(page.locator('.ep').first()).toBeVisible({ timeout: 20_000 });
|
||||
});
|
||||
|
||||
// Every row says "Admin" on its checkbox, so match the name exactly.
|
||||
const userRow = (page, name) =>
|
||||
page.locator('#modalCard [data-id]').filter({ has: page.locator('b', { hasText: new RegExp(`^${name}$`) }) });
|
||||
|
||||
async function openUsers(page) {
|
||||
await page.locator('#prefs').click();
|
||||
await page.locator('#gusers').click();
|
||||
await expect(userRow(page, 'admin')).toBeVisible();
|
||||
}
|
||||
|
||||
test('an admin adds someone, makes them an admin, and removes them', async ({ page }) => {
|
||||
await openUsers(page);
|
||||
await page.locator('#uname').fill('pat');
|
||||
await page.locator('#upass').fill('patpassword');
|
||||
await page.locator('#uadd').click();
|
||||
const row = userRow(page, 'pat');
|
||||
await expect(row).toBeVisible();
|
||||
// When each account was added and last signed in; the admin signed in with the token link.
|
||||
await expect(row).toContainText(/Added .* never signed in/);
|
||||
await expect(userRow(page, 'admin')).toContainText(/signed in \d+m ago/);
|
||||
await expect(row.locator('[data-a="admin"]')).not.toBeChecked();
|
||||
|
||||
await row.locator('[data-a="admin"]').check();
|
||||
// Not just the box: it has to have reached the database.
|
||||
await expect.poll(async () =>
|
||||
(await (await page.request.get('/api/users')).json()).find(u => u.name === 'pat')?.admin
|
||||
).toBe(true);
|
||||
|
||||
page.once('dialog', d => d.accept());
|
||||
await row.locator('[data-a="rm"]').click();
|
||||
await expect(row).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('the only admin cannot be demoted or removed', async ({ page }) => {
|
||||
await openUsers(page);
|
||||
await userRow(page, 'admin').locator('[data-a="admin"]').click();
|
||||
await expect(page.locator('.toast.bad')).toContainText('only admin');
|
||||
// Redrawn from the server, so the box is back.
|
||||
await expect(userRow(page, 'admin').locator('[data-a="admin"]')).toBeChecked();
|
||||
|
||||
const me = (await (await page.request.get('/api/users')).json()).find(u => u.name === 'admin');
|
||||
expect((await page.request.delete(`/api/users/${me.id}`)).status()).toBe(400);
|
||||
});
|
||||
|
||||
test('Settings exports your OPML and imports a pasted one', async ({ page }) => {
|
||||
await page.locator('#prefs').click();
|
||||
const [dl] = await Promise.all([
|
||||
page.waitForEvent('download'),
|
||||
page.locator('#modalCard a[title="Export OPML"]').click(),
|
||||
]);
|
||||
expect(dl.suggestedFilename()).toBe('ipx-subscriptions.opml');
|
||||
const out = require('fs').readFileSync(await dl.path(), 'utf8');
|
||||
for (const f of ['show.xml', 'pics.xml', 'multi.xml', 'subs.opml']) expect(out).toContain(f);
|
||||
// A feed from an OPML subscription comes back with the OPML itself, not on its own.
|
||||
expect(out).not.toContain('other.xml');
|
||||
|
||||
// One feed new to everyone, one the admin already has.
|
||||
await page.locator('#gopml').click();
|
||||
await page.locator('#opmlText').fill('<opml version="2.0"><head><title>t</title></head><body>' +
|
||||
'<outline text="Imported Show" xmlUrl="http://127.0.0.1:8792/imported.xml"/>' +
|
||||
'<outline text="Test Show" xmlUrl="http://127.0.0.1:8792/show.xml"/></body></opml>');
|
||||
await page.locator('#oimp').click();
|
||||
await expect(page.locator('.toast', { hasText: 'Subscribed to' }))
|
||||
.toHaveText('Subscribed to 1 feed(s), 1 you already had');
|
||||
// Named from the OPML's id until the first scan reads the feed's own title.
|
||||
await expect(page.locator('#feedlist .feed', { hasText: /Imported Show|imported-show/ }))
|
||||
.toBeVisible({ timeout: 20_000 });
|
||||
});
|
||||
|
||||
test('an uploaded OPML file imports, and a file that is not OPML is refused', async ({ page }) => {
|
||||
await page.locator('#prefs').click();
|
||||
await page.locator('#gopml').click();
|
||||
const pick = page.locator('#opmlFile');
|
||||
|
||||
// An RSS feed is XML but not OPML: refused in the page, and the picker lets go of it.
|
||||
await pick.setInputFiles({
|
||||
name: 'feed.xml', mimeType: 'application/xml',
|
||||
buffer: require('fs').readFileSync(require('path').join(__dirname, 'fixtures', 'show.xml')),
|
||||
});
|
||||
await page.locator('#oimp').click();
|
||||
await expect(page.locator('.toast.bad', { hasText: 'feed.xml is not an OPML file' })).toBeVisible();
|
||||
expect(await pick.evaluate(i => i.files.length)).toBe(0);
|
||||
|
||||
// Something that gets past the page's quick look is still refused by the server, untouched.
|
||||
const sneaky = await page.request.post('/api/opml', {
|
||||
data: { xml: '<rss version="2.0"><channel><title><opml></title></channel></rss>' },
|
||||
});
|
||||
expect(sneaky.status()).toBe(400);
|
||||
expect(await sneaky.text()).toContain('not an OPML file');
|
||||
|
||||
// A real one. Multi Show is already the admin's, so it counts as already had.
|
||||
await pick.setInputFiles({
|
||||
name: 'subs.opml', mimeType: 'text/x-opml',
|
||||
buffer: Buffer.from('<opml version="2.0"><head><title>t</title></head><body>' +
|
||||
'<outline text="Multi Show" xmlUrl="http://127.0.0.1:8792/multi.xml"/></body></opml>'),
|
||||
});
|
||||
await page.locator('#oimp').click();
|
||||
await expect(page.locator('.toast', { hasText: 'Subscribed to' }))
|
||||
.toHaveText('Subscribed to 0 feed(s), 1 you already had');
|
||||
await expect(page.locator('#modal.on')).toBeHidden();
|
||||
});
|
||||
|
||||
test('an export from one account imports into another', async ({ page, browser }) => {
|
||||
// Regression: import only added URLs the catalogue lacked and subscribed nobody, so importing
|
||||
// the admin's export into a second account did nothing at all.
|
||||
const { execFileSync } = require('child_process');
|
||||
const setup = require('./global-setup');
|
||||
const env = {
|
||||
...process.env,
|
||||
IPX_CONFIG: `${setup.root}/config/config.toml`,
|
||||
IPX_DATA_DIR: `${setup.root}/data`,
|
||||
};
|
||||
try {
|
||||
execFileSync('./target/debug/ipx', ['user', 'add', 'opal'], { input: 'opalpassword', env });
|
||||
} catch (e) {
|
||||
if (!String(e.stderr || e.stdout).includes('already exists')) throw e;
|
||||
}
|
||||
const ctx = await browser.newContext();
|
||||
const opal = await ctx.newPage();
|
||||
await opal.goto('/login');
|
||||
await opal.locator('#name').fill('opal');
|
||||
await opal.locator('#pw').fill('opalpassword');
|
||||
await opal.locator('button[type=submit]').click();
|
||||
await expect(opal.locator('#feedlist')).toContainText('No feeds.');
|
||||
|
||||
// Export used to hand anyone the whole catalogue. Opal has nothing yet, so gets nothing.
|
||||
const empty = await opal.request.get('/api/opml');
|
||||
expect(empty.status()).toBe(200);
|
||||
expect(await empty.text()).not.toContain('xmlUrl');
|
||||
|
||||
const urlsIn = xml => [...xml.matchAll(/xmlUrl="([^"]+)"/g)].map(m => m[1]).sort();
|
||||
const exported = await (await page.request.get('/api/opml')).text(); // the admin's
|
||||
const urls = urlsIn(exported);
|
||||
expect(urls.length).toBeGreaterThan(2);
|
||||
expect(await (await opal.request.post('/api/opml', { data: { xml: exported } })).json())
|
||||
.toEqual({ added: urls.length, already: 0 });
|
||||
expect(await (await opal.request.post('/api/opml', { data: { xml: exported } })).json())
|
||||
.toEqual({ added: 0, already: urls.length });
|
||||
|
||||
await opal.reload();
|
||||
await expect(opal.locator('.feed', { hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||
// The round trip closes: opal's own export now lists what the admin's did.
|
||||
expect(urlsIn(await (await opal.request.get('/api/opml')).text())).toEqual(urls);
|
||||
await ctx.close();
|
||||
});
|
||||
|
||||
test('ipx import subscribes the admin, and ipx export writes the feeds out', async () => {
|
||||
// Its own config and database. The CLI works in-process, and the suite's running daemon
|
||||
// reads config.toml once at start, so it would not see what the CLI added anyway.
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { execFileSync } = require('child_process');
|
||||
const setup = require('./global-setup');
|
||||
const dir = path.join(setup.root, 'cli');
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
fs.mkdirSync(path.join(dir, 'data'), { recursive: true });
|
||||
fs.writeFileSync(path.join(dir, 'config.toml'),
|
||||
`[general]\ndownload_dir = "${dir}/downloads"\nsocket = "${dir}/ipx.sock"\n`);
|
||||
const env = { ...process.env, IPX_CONFIG: path.join(dir, 'config.toml'), IPX_DATA_DIR: path.join(dir, 'data') };
|
||||
const ipx = (args, input) => execFileSync('./target/debug/ipx', args, { env, input, encoding: 'utf8' });
|
||||
|
||||
ipx(['user', 'add', 'boss'], 'bosspassword'); // the first account is the admin
|
||||
const opml = path.join(dir, 'in.opml');
|
||||
fs.writeFileSync(opml, '<opml version="2.0"><head><title>t</title></head><body>' +
|
||||
'<outline text="One" xmlUrl="http://127.0.0.1:8792/one.xml"/>' +
|
||||
'<outline text="Two" xmlUrl="http://127.0.0.1:8792/two.xml"/></body></opml>');
|
||||
expect(ipx(['import', opml])).toContain('subscribed boss to 2 feed(s); 0 already there');
|
||||
expect(ipx(['import', opml])).toContain('subscribed boss to 0 feed(s); 2 already there');
|
||||
|
||||
const out = path.join(dir, 'out.opml');
|
||||
ipx(['export', out]);
|
||||
const xml = fs.readFileSync(out, 'utf8');
|
||||
expect(xml).toContain('http://127.0.0.1:8792/one.xml');
|
||||
expect(xml).toContain('http://127.0.0.1:8792/two.xml');
|
||||
});
|
||||
|
||||
test('Popular lists what everyone here reads, but never a private feed', async ({ browser }) => {
|
||||
const { execFileSync } = require('child_process');
|
||||
const setup = require('./global-setup');
|
||||
const env = {
|
||||
...process.env,
|
||||
IPX_CONFIG: `${setup.root}/config/config.toml`,
|
||||
IPX_DATA_DIR: `${setup.root}/data`,
|
||||
};
|
||||
try {
|
||||
execFileSync('./target/debug/ipx', ['user', 'add', 'piper'], { input: 'piperpassword', env });
|
||||
} catch (e) {
|
||||
if (!String(e.stderr || e.stdout).includes('already exists')) throw e;
|
||||
}
|
||||
const ctx = await browser.newContext();
|
||||
const piper = await ctx.newPage();
|
||||
await piper.goto('/login');
|
||||
await piper.locator('#name').fill('piper');
|
||||
await piper.locator('#pw').fill('piperpassword');
|
||||
await piper.locator('button[type=submit]').click();
|
||||
await expect(piper.locator('#feedlist')).toContainText('No feeds.');
|
||||
|
||||
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
|
||||
const offered = piper.locator('#popular .childrow');
|
||||
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||
// An OPML is listed as the feeds inside it, and a key in a URL marks someone's paid feed.
|
||||
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
|
||||
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
|
||||
|
||||
// No URL reaches the page at all, so neither can a key, and the server holds the same line.
|
||||
const listed = await (await piper.request.get('/api/popular')).text();
|
||||
expect(listed).not.toContain('secret123');
|
||||
expect(listed).not.toContain('.xml');
|
||||
expect((await piper.request.post('/api/popular/paid-show')).status()).toBe(400);
|
||||
|
||||
// Popular is the top ten of the directory, and the directory is every listed feed A to Z,
|
||||
// with an OPML's feeds in place of the OPML in both.
|
||||
const dir = await (await piper.request.get('/api/directory')).json();
|
||||
const top = await (await piper.request.get('/api/popular')).json();
|
||||
const names = dir.map(p => (p.title || p.id).toLowerCase());
|
||||
expect(names).toEqual([...names].sort());
|
||||
const ids = dir.map(p => p.id);
|
||||
expect(ids).not.toContain('test-subscriptions');
|
||||
expect(ids).toEqual(expect.arrayContaining(['grouped-show', 'aardvark-radio']));
|
||||
expect(top.length).toBe(Math.min(10, dir.length));
|
||||
expect(top.every(t => ids.includes(t.id))).toBe(true);
|
||||
expect(ids).not.toContain('paid-show');
|
||||
|
||||
// Subscribe from the directory this time; the popular list shares the same rows.
|
||||
await piper.locator('#feedlist .place', { hasText: 'Directory' }).click();
|
||||
await expect(piper.locator('#count')).toContainText(`Directory: ${dir.length} feed`);
|
||||
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible();
|
||||
await expect(offered.filter({ hasText: /Grouped Show|grouped-show/ })).toBeVisible();
|
||||
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
|
||||
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
|
||||
|
||||
const row = async () =>
|
||||
(await (await piper.request.get('/api/popular')).json()).find(p => p.id === 'test-show');
|
||||
const before = await row();
|
||||
expect(before.subscribed).toBe(false);
|
||||
await offered.filter({ hasText: 'Test Show' }).locator('button[title="Subscribe"]').click();
|
||||
await expect(piper.locator('#feedlist .feed', { hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
// Everyone counts, you included: it stays listed, marked as yours, with one more subscriber.
|
||||
expect(await row()).toMatchObject({ subscribed: true, subscribers: before.subscribers + 1 });
|
||||
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
|
||||
await expect(offered.filter({ hasText: 'Test Show' }).locator('[title^="Subscribed"]')).toBeVisible();
|
||||
await expect(offered.filter({ hasText: 'Test Show' }).locator('button[title="Subscribe"]')).toHaveCount(0);
|
||||
|
||||
// All Subscriptions is every item from piper's feeds and only those: the admin's Picture
|
||||
// Blog is not among them.
|
||||
await piper.locator('#feedlist .place', { hasText: 'All Subscriptions' }).click();
|
||||
const first = piper.locator('.ep', { hasText: 'First Episode' });
|
||||
await expect(first).toBeVisible({ timeout: 20_000 });
|
||||
await expect(first.locator('.fd')).toHaveText('Test Show');
|
||||
await expect(piper.locator('.ep', { hasText: 'An Article' })).toHaveCount(0);
|
||||
await expect(piper.locator('#count')).toContainText('All Subscriptions:');
|
||||
await ctx.close();
|
||||
});
|
||||
|
||||
test('adding a feed scans it straight away', async ({ page }) => {
|
||||
await page.locator('#addFeed').click();
|
||||
await page.locator('#nurl').fill('http://127.0.0.1:8792/fresh.xml');
|
||||
await page.locator('#nsave').click();
|
||||
// Nobody pressed Scan. The scheduler's tick is a minute, so this is the add scanning it.
|
||||
await expect(page.locator('.ep', { hasText: 'Fresh Ep' })).toBeVisible({ timeout: 10_000 });
|
||||
});
|
||||
|
||||
test('a deleted file looks as if it was never downloaded', async ({ page }) => {
|
||||
// Other people subscribe to Picture Blog by now, so both prompts come; take them.
|
||||
page.on('dialog', d => d.accept());
|
||||
await page.locator('.feed', { hasText: 'Picture Blog' }).click();
|
||||
const row = page.locator('.ep', { hasText: 'An Article' });
|
||||
await expect(row).toBeVisible({ timeout: 20_000 });
|
||||
await row.click();
|
||||
await page.locator('#files button[data-a="del"]').click();
|
||||
|
||||
// No "reaped", no chip at all: just the way to get it again.
|
||||
await expect(row.locator('.kind.here')).toHaveCount(0);
|
||||
await expect(row).not.toContainText(/reaped/i);
|
||||
await row.click();
|
||||
await expect(page.locator('#files')).not.toContainText(/reaped/i);
|
||||
await expect(page.locator('#files button[title="Download to the server"]')).toBeVisible();
|
||||
});
|
||||
|
||||
test('one action, one icon: the toolbar, the page and every dialog agree', async ({ page }) => {
|
||||
const icon = loc => loc.locator('svg path').first().getAttribute('d');
|
||||
await page.locator('#feedlist .feed', { hasText: 'Test Show' }).first().click();
|
||||
|
||||
// Unsubscribe is a minus in the toolbar and the feed header, never the x that closes things.
|
||||
expect(await icon(page.locator('#content .acts [data-a="rm"]'))).toBe(await icon(page.locator('#tbRemove')));
|
||||
|
||||
// The toolbar's read and keep show the selected item's state, as its own buttons do, and follow
|
||||
// a change made from the toolbar.
|
||||
await page.locator('.ep').first().click();
|
||||
const pair = async a => [await icon(page.locator(a === 'read' ? '#tbRead' : '#tbFlag')),
|
||||
await icon(page.locator(`#detail [data-a="${a}"]`))];
|
||||
for (const a of ['read', 'flag']) { const [tb, own] = await pair(a); expect(tb).toBe(own); }
|
||||
const [kept] = await pair('flag');
|
||||
await page.locator('#tbFlag').click();
|
||||
await expect.poll(async () => { const [tb, own] = await pair('flag'); return tb === own && tb !== kept; }).toBe(true);
|
||||
await page.locator('#tbFlag').click(); // leave it as it was
|
||||
await expect.poll(async () => (await pair('flag'))[0]).toBe(kept);
|
||||
|
||||
// Every button in every dialog is an icon with its words in the tooltip.
|
||||
const dialogs = [
|
||||
() => page.locator('#addFeed').click(),
|
||||
() => page.locator('#prefs').click(),
|
||||
async () => { await page.locator('#prefs').click(); await page.locator('#gusers').click(); },
|
||||
async () => { await page.locator('#prefs').click(); await page.locator('#gopml').click(); },
|
||||
() => page.locator('#logs').click(),
|
||||
() => page.locator('#content .acts [data-a="settings"]').click(),
|
||||
() => page.locator('#content .acts [data-a="dl"]').click(),
|
||||
() => page.locator('#content .acts [data-a="rm"]').click(),
|
||||
];
|
||||
for (const open of dialogs) {
|
||||
await open();
|
||||
const btns = page.locator('#modalCard .btn');
|
||||
await expect(btns.first()).toBeVisible();
|
||||
for (const b of await btns.all()) {
|
||||
await expect(b.locator('svg')).toHaveCount(1);
|
||||
await expect(b).toHaveAttribute('title', /\S/);
|
||||
}
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(page.locator('#modal.on')).toBeHidden();
|
||||
}
|
||||
});
|
||||
|
||||
test('All Subscriptions marks everything read, across every feed', async ({ page }) => {
|
||||
const all = page.locator('#feedlist .place', { hasText: 'All Subscriptions' });
|
||||
await all.click();
|
||||
// Earlier tests read things; make sure something is unread. Opening an item reads it, and
|
||||
// its own button makes it unread again.
|
||||
await page.locator('.ep').first().click();
|
||||
await page.locator('#detail [data-a="read"][title="Mark unread"]').click();
|
||||
await expect(all.locator('.badge')).not.toHaveText('0');
|
||||
|
||||
page.once('dialog', d => d.accept());
|
||||
await page.locator('#content .acts [data-a="readall"]').click();
|
||||
await expect(all.locator('.badge')).toHaveText('0');
|
||||
await page.locator('.tabs button', { hasText: 'Unread' }).click();
|
||||
await expect(page.locator('.ep')).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('the item table sorts by any column, both ways, and remembers', async ({ page }) => {
|
||||
const all = page.locator('#feedlist .place', { hasText: 'All Subscriptions' });
|
||||
await all.click();
|
||||
const head = k => page.locator(`#list .ephead [data-sort="${k}"]`);
|
||||
const titles = () => page.locator('#eps .ep .t').allTextContents();
|
||||
// Byte order on lower case, which is what SQLite gives for lower(...).
|
||||
const cmp = (a, b) => (a.toLowerCase() < b.toLowerCase() ? -1 : a.toLowerCase() > b.toLowerCase() ? 1 : 0);
|
||||
const sorted = (t, dir) => JSON.stringify(t) === JSON.stringify([...t].sort((a, b) => cmp(a, b) * dir));
|
||||
await expect(page.locator('#eps .ep').nth(2)).toBeVisible({ timeout: 20_000 });
|
||||
expect(new Set(await titles()).size).toBeGreaterThan(2); // or both orders would prove nothing
|
||||
|
||||
await expect(head('title')).toHaveText('Title');
|
||||
await head('title').click();
|
||||
await expect.poll(async () => sorted(await titles(), 1)).toBe(true);
|
||||
await head('title').click();
|
||||
await expect.poll(async () => sorted(await titles(), -1)).toBe(true);
|
||||
|
||||
// Kept across a reload.
|
||||
await page.reload();
|
||||
await all.click();
|
||||
await expect(head('title').locator('.arr.desc')).toBeVisible();
|
||||
await expect.poll(async () => sorted(await titles(), -1)).toBe(true);
|
||||
|
||||
// Size has its own column; the file column is just what the file is.
|
||||
await expect(page.locator('#eps .ep .size', { hasText: /\d/ }).first()).toBeVisible();
|
||||
await expect(page.locator('#eps .ep .file', { hasText: /\d/ })).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('play in the Files pane plays once, in the player bar', async ({ page }) => {
|
||||
// Regression: the pane had an <audio> of its own, and playing it started the player bar too,
|
||||
// so the same file played twice at once.
|
||||
await page.locator('#feedlist .feed', { hasText: 'Test Show' }).first().click();
|
||||
await page.locator('.ep', { has: page.locator('.kind.here') }).first().click();
|
||||
await page.locator('#files [data-a="play"]').click();
|
||||
await expect(page.locator('#player')).toBeVisible();
|
||||
await expect(page.locator('audio')).toHaveCount(1); // the player bar's, and nothing else
|
||||
await page.locator('#pclose').click();
|
||||
});
|
||||
|
||||
test('someone the proxy signs in never sees the password page, and signs out through the proxy', async ({ page, browser }) => {
|
||||
// Signed in with the token, not by the proxy: Sign out stays ipx's own.
|
||||
expect((await (await page.request.get('/api/me')).json()).sign_out).toBeNull();
|
||||
|
||||
const ctx = await browser.newContext({ extraHTTPHeaders: { 'X-Test-User': 'proxied@example.com' } });
|
||||
const proxied = await ctx.newPage();
|
||||
// Regression: after Sign out, the password form showed to someone the proxy still vouched for.
|
||||
await proxied.goto('/login');
|
||||
await expect(proxied).toHaveURL(/:8791\/$/);
|
||||
await expect(proxied.locator('#who')).toContainText('proxied@example.com');
|
||||
expect(await (await proxied.request.get('/api/me')).json())
|
||||
.toMatchObject({ name: 'proxied@example.com', sign_out: '/signed-out-by-the-proxy' });
|
||||
await proxied.locator('#signout').click();
|
||||
await expect(proxied).toHaveURL(/\/signed-out-by-the-proxy$/);
|
||||
await ctx.close();
|
||||
});
|
||||
|
||||
5
tests/ui/fixtures/aardvark.xml
Normal file
5
tests/ui/fixtures/aardvark.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Aardvark Radio</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Inside the OPML, and first in it and alphabetically.</description>
|
||||
<item><title>Aardvark Ep</title><guid>aa-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
5
tests/ui/fixtures/fresh.xml
Normal file
5
tests/ui/fixtures/fresh.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Fresh Show</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Added in the browser suite, and scanned by adding it.</description>
|
||||
<item><title>Fresh Ep</title><guid>fresh-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
5
tests/ui/fixtures/imported.xml
Normal file
5
tests/ui/fixtures/imported.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Imported Show</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Only ever arrives through an OPML import.</description>
|
||||
<item><title>Imported Ep</title><guid>imp-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
5
tests/ui/fixtures/paid.xml
Normal file
5
tests/ui/fixtures/paid.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Paid Show</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Subscribed with a key in its URL, so it must never be offered to anyone else.</description>
|
||||
<item><title>Paid Ep</title><guid>paid-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
@@ -1,4 +1,5 @@
|
||||
<opml version="2.0"><head><title>Test Subscriptions</title></head>
|
||||
<body><outline text="Folder">
|
||||
<outline type="rss" text="Aardvark Radio" xmlUrl="http://127.0.0.1:8792/aardvark.xml"/>
|
||||
<outline type="rss" text="Grouped Show" xmlUrl="http://127.0.0.1:8792/other.xml"/>
|
||||
</outline></body></opml>
|
||||
</outline></body></opml>
|
||||
|
||||
@@ -10,7 +10,15 @@ const TOKEN = 'testtokentesttokentesttoken12345'; // fixed, so tests need not
|
||||
// Called from playwright.config.js at load time, NOT as globalSetup: Playwright starts
|
||||
// webServer *before* globalSetup, so a config written there does not exist yet when the
|
||||
// daemon launches -- it would fall back to the real config and fight the live daemon.
|
||||
// Playwright imports this config again in every worker process, so prepare() runs more
|
||||
// than once per suite. Wiping on the second call deleted the data directory out from under
|
||||
// the running daemon: it kept serving from the unlinked inode, while anything else opening
|
||||
// that path -- the CLI, a query -- got a brand new empty database and disagreed with it.
|
||||
function prepare() {
|
||||
// Only the process that launches the run may wipe. A worker gets TEST_WORKER_INDEX.
|
||||
if (process.env.TEST_WORKER_INDEX !== undefined || process.env.PW_WORKER_INDEX !== undefined) {
|
||||
return;
|
||||
}
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
for (const d of ['config', 'data', 'downloads']) {
|
||||
fs.mkdirSync(path.join(root, d), { recursive: true });
|
||||
@@ -29,6 +37,10 @@ enabled = false
|
||||
enabled = true
|
||||
bind = "127.0.0.1:8791"
|
||||
token = "${TOKEN}"
|
||||
# The proxy path, for tests that send the header themselves: the daemon sees them at 127.0.0.1.
|
||||
trusted_header = "X-Test-User"
|
||||
trusted_proxies = ["127.0.0.1"]
|
||||
sign_out_url = "/signed-out-by-the-proxy"
|
||||
|
||||
[feeds.test-show]
|
||||
url = "http://127.0.0.1:8792/show.xml"
|
||||
@@ -47,6 +59,11 @@ auto_download = true
|
||||
[feeds.test-subscriptions]
|
||||
url = "http://127.0.0.1:8792/subs.opml"
|
||||
auto_download = false
|
||||
|
||||
# A key in its URL, like a Patreon feed: someone's paid subscription, never offered to others.
|
||||
[feeds.paid-show]
|
||||
url = "http://127.0.0.1:8792/paid.xml?auth=secret123"
|
||||
auto_download = false
|
||||
`);
|
||||
}
|
||||
|
||||
|
||||
1186
web/index.html
1186
web/index.html
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user