Compare commits
89 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| be3820bbbd | |||
| 51ce0bf9eb | |||
| 6ec900e456 | |||
| d4304869b6 | |||
| 9aae3097e7 | |||
| 2ff2074755 | |||
| 1698cf8d1e | |||
| b94a74ef15 | |||
| 9a8a3c696f | |||
| bedf64e645 | |||
| 586d2c07a1 | |||
| 2ba83c3aed | |||
| 1352f0d54d | |||
| a958f7cb37 | |||
| 457a58dcc5 | |||
| 2af57065c6 | |||
| 564b011c7a | |||
| 9eb7aadced | |||
| c6bceaef37 | |||
| dc63d6acaf | |||
| 8937f35f00 | |||
| 0990f2a90d | |||
| 0cc002cdfa | |||
| 2c9e899762 | |||
| 2e416f96cf | |||
| 9269aa99f7 | |||
| ae8123250b | |||
| 2c34a144ba | |||
| 669e8b5124 | |||
| 5362436766 | |||
| 0efc49519c | |||
| 57dcba2d1a | |||
| e95cccc66f | |||
| 7295be8b25 | |||
| f1b0d97b81 | |||
| ae47e31a97 | |||
| 8a309eb652 | |||
| 1a2b0d87c6 | |||
| d7fac2d0e7 | |||
| df9b7645d6 | |||
| f3825cfc57 | |||
| c0f4b0bcb2 | |||
| 8b8b48302b | |||
| 5d3fdde4da | |||
| 8784d0a3fd | |||
| 5e95557cbb | |||
| 6114add4a6 | |||
| c47c224372 | |||
| 686851b448 | |||
| f0d03c79c8 | |||
| 7df4ee7dde | |||
| d46ec73261 | |||
| 4810bb5cfb | |||
| 06f182b555 | |||
| ed26fa2061 | |||
| 772dda5154 | |||
| 3817dbebdd | |||
| 3ee65f3155 | |||
| d157c88ba9 | |||
| 7473d5b7fb | |||
| 8f5e2749ff | |||
| dfcdf47143 | |||
| 470f3e1ff1 | |||
| c77d152015 | |||
| 5190a29cdb | |||
| dd9c0881ee | |||
| 6d6b4dd1e4 | |||
| ddb9dbb7e1 | |||
| a83f62ccd3 | |||
| 8b21d19365 | |||
| 4429f1119c | |||
| 665d5b8ecb | |||
| c86d698363 | |||
| 5f6e2a8dc1 | |||
| 19309d609f | |||
| e97f2b9c2f | |||
| 9960befed5 | |||
| 9dc4c1ddfa | |||
| 9c5e7716c5 | |||
| cbd5e68d13 | |||
| e05ea62051 | |||
| aecad18d22 | |||
| 44623098ae | |||
| 72280b9ccd | |||
| 5666166769 | |||
| 93b4815d84 | |||
| 74ec6e9281 | |||
| ed47e456d4 | |||
| c12e8ca19c |
4
.dockerignore
Normal file
4
.dockerignore
Normal file
@@ -0,0 +1,4 @@
|
||||
target/
|
||||
.git/
|
||||
*.md
|
||||
tests/
|
||||
3
.gitignore
vendored
3
.gitignore
vendored
@@ -1 +1,4 @@
|
||||
/target
|
||||
/node_modules
|
||||
/test-results
|
||||
/playwright-report
|
||||
|
||||
343
CHANGELOG.md
Normal file
343
CHANGELOG.md
Normal file
@@ -0,0 +1,343 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
The long form, with what was wrong before and how it was found, is in
|
||||
[docs/history.md](docs/history.md).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.5.3] - 2026-09-14
|
||||
|
||||
### Added
|
||||
|
||||
- A feed that has been failing for a day shows a plain-English reason in the sidebar and on its
|
||||
own page, sorted from a 404, a 401/403, a 402, a name that no longer resolves, or a web page in
|
||||
place of the feed -- with Unsubscribe or, when the page links its new feed, Use the new address.
|
||||
A feed that fails once and reads fine again within a day is never flagged.
|
||||
|
||||
### Changed
|
||||
|
||||
- Unsubscribing from the last person's OPML or Patreon subscription now retires the feeds it
|
||||
listed, the same as a feed the list itself drops: removed if nothing was downloaded, kept and
|
||||
marked orphaned otherwise. Until now they stayed in the database and kept being scanned hourly
|
||||
with auto-download on, which is how 922 defunct `davewiner` feeds outlived the OPML that listed
|
||||
them.
|
||||
|
||||
### Fixed
|
||||
|
||||
- A feed whose XML uses a bare `&` instead of `&` (kcpw, both feedland feeds) is now read
|
||||
instead of refused.
|
||||
- A feed URL that now serves a web page says so, and names the feed the page links to when it has
|
||||
one, instead of a raw XML parser error.
|
||||
- A publisher answering with an empty body (British Antarctic Survey's 202) is read as nothing new
|
||||
to report, not a parse failure.
|
||||
- A link in an item's show notes opens in a new tab instead of navigating away from ipx.
|
||||
- A video file plays as video, in a small floating pane above the player bar, instead of silently
|
||||
as sound only.
|
||||
- On the Unread tab, opening an item no longer makes it disappear from the list -- it stays until
|
||||
you open a different one, even if a scan finishes and refreshes the list while it is open.
|
||||
- Subscribe and Unsubscribe have their own icons (a circled check and a circled minus) instead of
|
||||
sharing the generic plus and minus used for adding feeds, users and imports.
|
||||
- Settings no longer disappears for a non-admin account. It was hiding the whole Settings modal
|
||||
along with the log and the users screen, but a non-admin has settings of their own in there --
|
||||
their subscriptions' Export and Import, and the schedule and quota are worth seeing even without
|
||||
a say in them. Only the log and the users screen, which the server also refuses them, are gone.
|
||||
|
||||
## [0.5.2] - 2026-09-12
|
||||
|
||||
### Added
|
||||
|
||||
- Settings → Users and `ipx user list` show when each account was added and when it last signed
|
||||
in, to the hour.
|
||||
- `ipx user rename <name> <new name>` renames an account and keeps its feeds, read state and admin
|
||||
rights. An account made before the proxy was set up can take the name the proxy signs it in as.
|
||||
|
||||
### Changed
|
||||
|
||||
- Directory and Popular list the feeds inside an OPML one by one, and no longer the OPML itself,
|
||||
so you can subscribe to just the shows you want.
|
||||
- The database no longer records when subscriptions and sign-in sessions were created. Nothing
|
||||
ever read it, and an existing database drops the columns on its next start.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Show notes that the podcast's host cut off in the middle of a tag no longer open with a scrap of
|
||||
HTML: the item's other copy of its notes is used instead, from the next time the feed changes.
|
||||
Daily Meditation Podcast had 57.
|
||||
- Docker no longer shows ipodderx as starting, or calls it unhealthy, while it scans or downloads:
|
||||
`ipx status` answers at once instead of waiting for the job in progress to finish.
|
||||
- Signing out after signing in through Cloudflare Access no longer lands on ipodderx's own password
|
||||
page. With the new `sign_out_url` set, Sign out ends the Access session, and the password page
|
||||
sends anyone the proxy signs in straight to their feeds.
|
||||
- The sign-in guide, `docs/sso.md`, describes the setup ipodderx.sdf1.net really runs: Authentik as
|
||||
Cloudflare Access's identity provider, and how to find the address ipx has to trust. It had never
|
||||
been checked against a real setup, and pointed at the wrong address.
|
||||
|
||||
## [0.5.1] - 2026-09-12
|
||||
|
||||
### Fixed
|
||||
|
||||
- The triangle that opens an OPML or Patreon folder was cramped against the folder's art. It has
|
||||
more room now, and a wider target to click.
|
||||
|
||||
## [0.5.0] - 2026-09-12
|
||||
|
||||
### Added
|
||||
|
||||
- A Patreon token pasted into Add feed, or a creator's RSS link without `&show=`, becomes a folder
|
||||
of that creator's shows, kept in step on every scan like a subscribed OPML. A creator with only
|
||||
one show stays a plain feed. One already added as a single long feed is split into its shows on
|
||||
its next scan, keeping its files and what you had read.
|
||||
- Add feed has an "Allow items marked explicit" box, so a new feed's first scan no longer skips
|
||||
every explicit item.
|
||||
|
||||
### Changed
|
||||
|
||||
- Unread counts, unread dots and download progress are amber, the colour of the icon's EQ bars.
|
||||
Blue is kept for the primary action and links, so a count no longer looks like a button.
|
||||
- What is playing is marked by small EQ bars, in its row and in the player. They move only while it
|
||||
plays.
|
||||
- A folder in the sidebar shows its first four shows' art as a mosaic, and its shows sit under its
|
||||
title. Only folders have a triangle, so every feed lines up with Directory and Popular above.
|
||||
- Feeds without art get initials in a colour of their own, instead of all the same grey.
|
||||
- The Flagged tab is called Kept, as the Keep button and Settings already said.
|
||||
- A feed's header is one short line; when it checks next is in its tooltip.
|
||||
- The item list takes more of the window, and the Files pane shows only when the item has files.
|
||||
- Column headings and tags are in sentence case, and fewer things are bold.
|
||||
- Unsubscribe is a round button beside the feed's other actions.
|
||||
- Export and Import in Settings say what they do.
|
||||
- The sign-in page shows the original icon large.
|
||||
- Nothing animates when your system asks for reduced motion.
|
||||
- The pages are about 90 KB smaller: the icon is served once instead of written into each.
|
||||
- A web token generated for a new install is 64 characters instead of 32.
|
||||
- The README is a short overview of what ipx does and how to run it, and points into `docs/` for
|
||||
the rest. It still described the layout from before 0.4.0.
|
||||
|
||||
### Removed
|
||||
|
||||
- The systemd units in `contrib/`. Run ipx with Docker, or point a unit of your own at
|
||||
`ipx daemon`.
|
||||
- Upgrading from before 0.3.0 directly: what was read, kept or part-played before accounts is no
|
||||
longer carried over to the admin, and OPML feeds that old versions wrote into `config.toml` are
|
||||
no longer moved out of it. Upgrade through 0.4.0 first.
|
||||
- `interval_mins` in `config.toml` is ignored; use `schedule`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- The feed list works from the keyboard: Tab reaches every feed, Enter opens it, and Right and Left
|
||||
open and close a folder. Every button shows where the focus is, including in the toolbar, which
|
||||
used to clip the ring.
|
||||
- "1 items" reads "1 item".
|
||||
- A selected feed without art no longer loses its initials tile in Dark and Light.
|
||||
- The player shows the feed's initials when there is no art, not the episode's.
|
||||
- Turning on Allow explicit, or changing keywords or auto-download, brings back what those settings
|
||||
had skipped on the feed's next scan. Before, an item was judged once, when first seen, and a
|
||||
skipped one stayed skipped whatever you changed.
|
||||
- Feeds inside an OPML or a Patreon creator follow your settings on the folder unless you set their
|
||||
own, as the folder's settings dialog said they did. Before, the folder's settings reached nothing
|
||||
inside it.
|
||||
- A new feed no longer takes the name of one you removed earlier and shows that feed's old items.
|
||||
Re-adding the same feed still gets its old name, and its history, back.
|
||||
|
||||
## [0.4.0] - 2026-09-11
|
||||
|
||||
### Added
|
||||
|
||||
- A Classic theme after the 2004 Mac app, beside Dark and Light: brushed-metal toolbar, Aqua
|
||||
blue selection, red unread badges, a striped table and Lucida Grande. The theme button steps
|
||||
through all three and remembers the choice.
|
||||
- A toolbar across the top, after the original iPodderX: add and unsubscribe, play, mark read
|
||||
and keep for the selected item, scan, a search box for what is showing, and Settings and Log.
|
||||
- Directory, Popular and All Subscriptions at the top of the feed list, opening in the main pane.
|
||||
Directory lists every feed anyone here subscribes to, A to Z (`GET /api/directory`). All
|
||||
Subscriptions lists every item from every feed you subscribe to (`GET /api/entries`).
|
||||
- Items show as a table (unread, kept, title, feed, file, size, published) with a Files pane beside
|
||||
it, and a status bar with the totals.
|
||||
- Mark everything read from All Subscriptions, across every feed you subscribe to
|
||||
(`POST /api/read-all`). It asks first. All Subscriptions can also check every feed from its header.
|
||||
- Click a column heading in the item table to sort by it (kept, title, feed, file type, size,
|
||||
published); click again to reverse. The server sorts, so it covers the whole list, not just the
|
||||
fifty shown, and the choice is remembered.
|
||||
|
||||
### Changed
|
||||
|
||||
- Popular shows the top 10, not 20, and counts everyone, you included. Your own feeds stay on it,
|
||||
marked Subscribed, and clicking one opens it.
|
||||
- Adding a feed scans it straight away, and an OPML import that added feeds scans them, so their
|
||||
items show without pressing Scan.
|
||||
- A file deleted to save space, or by hand, looks as if it was never downloaded: no "reaped"
|
||||
label, just the Download button. The retention summary says "deleted", not "reaped".
|
||||
- Buttons are icons, with the words in their tooltips: the Files pane (save, delete, view,
|
||||
download), an item's own buttons (mark read, keep, open the original), the feed header (scan,
|
||||
download latest, mark all read, settings, unsubscribe), and the Settings, feed settings and
|
||||
Download latest dialogs (save, download, cancel). The icons are Font Awesome Free, embedded as
|
||||
SVG: only the ones used, no font to download, and nothing fetched from anyone else. They
|
||||
replace font characters such as ⟳ ⤓ ↗, which came out thin and tiny and differed from font to
|
||||
font. Keep is a flag everywhere, as it was in the original, and mark unread is an envelope.
|
||||
- One meaning per icon. Minus unsubscribes, x closes or cancels, plus adds or subscribes, and a
|
||||
dialog's confirm button carries the icon of what it does. The feed header's unsubscribe was an x
|
||||
and read as closing the page. The remaining word buttons are icons too:
|
||||
- Log, Add feed, Users, Unsubscribe and OPML.
|
||||
- Popular's Subscribe, Copy and Sign out.
|
||||
- The Subscribed label in Popular, the Directory and Add feed, which is now a green check.
|
||||
- The player's back, play, forward and close, which were font characters, and the folder arrow.
|
||||
- The toolbar's read and keep buttons show the selected item's state, with the same icons as the
|
||||
item's own buttons. Play, read and keep sit together, and Scan sits with add and unsubscribe.
|
||||
- An OPML subscription's page has the same header as a feed's, with its buttons in the same places.
|
||||
- The item table's size has its own column, apart from the file's type, and shows KB for small
|
||||
files instead of "0 MB". The Item heading is now Title.
|
||||
- A file's type is an icon (audio, video, image, PDF, torrent, other), green once it is
|
||||
downloaded and red when the download failed, with the details in its tooltip. One icon per
|
||||
row keeps the column lined up. The DOWNLOADED and PENDING labels are gone.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Playing a file from the Files pane played it twice at once, in the pane and in the player bar.
|
||||
The pane has a play button now, and the player bar is the only player.
|
||||
- The password box in Manage users was white in the dark theme.
|
||||
- An item with no date showed a stray dot in its details.
|
||||
- Escape did not close a dialog while the cursor was in one of its boxes, so Add feed, which opens
|
||||
in its URL box, could not be closed with Escape.
|
||||
|
||||
### Security
|
||||
|
||||
- Feeds from paid-feed services (Patreon, Supercast, Supporting Cast, Glow, Memberful) are never
|
||||
listed in Popular or the Directory. A Supercast feed, which keeps its key in the URL's path
|
||||
rather than the query, was being listed.
|
||||
|
||||
## [0.3.0] - 2026-09-11
|
||||
|
||||
### Added
|
||||
|
||||
- Add feed lists what other people on this server subscribe to, most subscribers first, and
|
||||
subscribes you by id (`GET /api/popular`, `POST /api/popular/{id}`). Feeds from an OPML, and
|
||||
feeds with a login or a key in their URL, are never listed.
|
||||
- Upload an OPML file to import, beside the paste box. The page checks it looks like OPML before
|
||||
sending it and clears the picker afterwards.
|
||||
- Settings → Manage users: add and remove accounts, and choose who is an admin
|
||||
(`GET`/`POST /api/users`, `PATCH`/`DELETE /api/users/{id}`).
|
||||
- Per-user subscriptions, and per-user read, starred and playback state. The existing library is
|
||||
adopted by the admin on first start.
|
||||
- Subscribing to a feed someone else already has costs no second fetch and no second copy. Scanning
|
||||
merges every subscriber's wants.
|
||||
- Delete on a shared feed reads **Delete for everyone**, and the server answers `409` while anyone
|
||||
else has starred the item or not played it (`?force=true` overrides).
|
||||
- A shared feed's header says how many other people read it.
|
||||
- `docs/` for configuration, the CLI, users, SSO and architecture, and `CLAUDE.md` for anyone
|
||||
working on the code.
|
||||
- Browser tests for OPML import and export by every route, user admin, unread ordering, and
|
||||
`ipx import`/`ipx export`.
|
||||
|
||||
### Changed
|
||||
|
||||
- Feeds inside an OPML subscription list the ones with unread items first.
|
||||
- OPML import subscribes you to every feed in the file. `ipx import` subscribes the first admin.
|
||||
- OPML export lists only your own subscriptions.
|
||||
- Production runs as a Docker image pushed to `192.168.1.130:5000` and recreated with
|
||||
`docker compose`.
|
||||
- This changelog follows Keep a Changelog. The long-form entries moved to `docs/history.md`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Importing another account's OPML export subscribed nobody and reported "Imported 0 feed(s)".
|
||||
Feeds it added had no subscriber, so they were never scanned.
|
||||
- Importing something that is not OPML answered `500`. It is now `400` "that is not an OPML file",
|
||||
refused before anything changes.
|
||||
- Starring stopped protecting a file from the quota and age sweeps once read state became per-user.
|
||||
|
||||
### Security
|
||||
|
||||
- The log is admin-only (`GET /api/logs` answers `403`, and the Log button is hidden). It names
|
||||
every account, every feed and every failed sign-in.
|
||||
- OPML export no longer hands anyone signed in the whole catalogue, including other people's
|
||||
private feed URLs.
|
||||
|
||||
## [0.2.0] - 2026-09-10
|
||||
|
||||
### Added
|
||||
|
||||
- Web UI served by the daemon: plain HTML and JS compiled into the binary, with feeds, items,
|
||||
filters, search, sanitised show notes and live progress over SSE.
|
||||
- Player bar with resume, speed, keyboard shortcuts and lock-screen controls.
|
||||
- Three-pane layout: feeds beside, items above, and the selected item's text and files below.
|
||||
- Phone layout.
|
||||
- Accounts and sign-in: Argon2id passwords, session cookies, `ipx user add|list|passwd|rm`, and a
|
||||
trusted proxy header for Cloudflare Zero Trust or Authentik (`docs/sso.md`).
|
||||
- Subscribing to an OPML: it is re-read every scan and its feeds show as a folder. A feed dropped
|
||||
from it is removed unless it has downloads.
|
||||
- Scheduling: a global interval with per-feed overrides (`every 30m`, `4h`, `1d`, `2w`).
|
||||
- `[general] media_types`, default audio and video, and `max_new_per_check`, default 3.
|
||||
- Every enclosure of an item, a View link for files that are not audio or video, and per-item
|
||||
artwork.
|
||||
- In-app log view with Daemon I/O, Scans and HTTP tabs.
|
||||
- Mark all read on an OPML subscription.
|
||||
- Editable feed URL with a copy button.
|
||||
- Docker image whose healthcheck goes through the control socket.
|
||||
- The iPodderX name, icon, and a colour scheme taken from the icon.
|
||||
- `tests/page-smoke.js` and a Playwright browser suite.
|
||||
|
||||
### Changed
|
||||
|
||||
- Global settings and scan schedules are admin-only. The per-feed schedule picker is gone.
|
||||
- Feeds from an OPML live in the database, not `config.toml`.
|
||||
- Opening an item marks it read. Playing it marks it read only at the end or past 90%.
|
||||
- "Episodes" became "items", since half the library is text.
|
||||
- A burst of scan events causes one refresh, not one per feed.
|
||||
- All is the default filter. OPML import and export, Settings and Log moved out of the header.
|
||||
- Torrents run detached, two at a time.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Download fetched the next queued episodes instead of the one clicked.
|
||||
- Pressing play made an item vanish from the Unread list.
|
||||
- Clearing a folder, schedule or cap from the UI did nothing.
|
||||
- The daemon ignored SIGTERM until the current download finished.
|
||||
- A missing function stopped the page script and left the whole UI dead.
|
||||
- One download painted progress on every pending row.
|
||||
- A torrent could freeze scanning for up to an hour.
|
||||
- Downloading from an OPML feed failed with "belongs to unsubscribed feed".
|
||||
- Image enclosures were downloaded, counted as episodes and given a play button.
|
||||
- A feed whose entries had been deleted stayed empty, because the server kept answering `304`.
|
||||
- An item with several enclosures kept only the last.
|
||||
- OPML folders rendered open by default.
|
||||
- Mark read in the text pane recursed until the stack overflowed.
|
||||
- The Unread, Downloaded and Flagged filters answered `500` without a search term.
|
||||
- An OPML subscription always showed 0 unread.
|
||||
- Folder names kept doubled spaces where separators were stripped.
|
||||
- Sidebar rows had four different left edges.
|
||||
|
||||
### Security
|
||||
|
||||
- The web UI needs a token or a sign-in. The token is compared in constant time, and an empty
|
||||
token refuses to serve.
|
||||
- Show notes are sanitised with `ammonia`.
|
||||
- A proxy's user header is honoured only from an address in `trusted_proxies`.
|
||||
- A feed URL must be http(s), so `file:///etc/passwd` is refused.
|
||||
- Download folders are sanitised per path segment, so `../../etc/Show` cannot climb out.
|
||||
|
||||
## [0.1.0] - 2026-09-09
|
||||
|
||||
### Added
|
||||
|
||||
- `ipx`, a Rust rewrite of the iPodderX engine: TOML config, SQLite state, and `ipx list`, `add`,
|
||||
`rm` and `fetch`.
|
||||
- RSS and Atom parsing with conditional GET, `<ttl>` and basic auth.
|
||||
- Streaming downloads with explicit, keyword and per-scan filters, deduplicated by enclosure URL.
|
||||
- Quota and age retention that never touches a starred file, and `ipx reap [--dry-run]`.
|
||||
- `ipx daemon` with a JSON-lines Unix socket. CLI commands proxy to a running daemon.
|
||||
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
|
||||
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
|
||||
|
||||
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.3...main
|
||||
[0.5.3]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.2...v0.5.3
|
||||
[0.5.2]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.1...v0.5.2
|
||||
[0.5.1]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.0...v0.5.1
|
||||
[0.5.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...v0.5.0
|
||||
[0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0
|
||||
[0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0
|
||||
[0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0
|
||||
[0.1.0]: https://git.sdf1.net/rays/ipodderx-rs/releases/tag/v0.1.0
|
||||
153
CLAUDE.md
Normal file
153
CLAUDE.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# Working on ipodderx-rs
|
||||
|
||||
Notes for whoever picks this up next. Read [docs/architecture.md](docs/architecture.md) for how the
|
||||
thing is built; this file is about working on it without repeating mistakes that have already been
|
||||
made here.
|
||||
|
||||
## Where things are
|
||||
|
||||
Production is the `iPodderX` container on Tower (192.168.1.130), the `ipodderx` service of the
|
||||
Arcane project `content`: `/mnt/fast/arcane/projects/content/compose.yaml`. That file is what runs;
|
||||
`docker-compose.yml` in this repo is a copy, and editing it changes nothing in production.
|
||||
|
||||
| | Host | In the container |
|
||||
|---|---|---|
|
||||
| Image | `192.168.1.130:5000/ipodderx:latest` | |
|
||||
| Config | `/mnt/fast/appdata/ipodderx/config.toml` | `/config/config.toml` |
|
||||
| Database | `/mnt/user/ipodderx/state.db` | `/data/state.db` |
|
||||
| Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` |
|
||||
| Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` |
|
||||
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | trusts `Cf-Access-Authenticated-User-Email` from `192.168.16.1`, the `content_default` gateway |
|
||||
|
||||
Deploying a change is: build and push the image, then pull it and recreate the container.
|
||||
|
||||
```sh
|
||||
docker buildx build --tag 192.168.1.130:5000/ipodderx:latest . --push
|
||||
docker compose -f /mnt/fast/arcane/projects/content/compose.yaml pull ipodderx
|
||||
docker compose -f /mnt/fast/arcane/projects/content/compose.yaml up -d ipodderx
|
||||
docker logs --tail 20 iPodderX
|
||||
```
|
||||
|
||||
**Name the service.** A bare `up -d` recreates every container in `content`, beets and immich
|
||||
included. Run `pull` before `up`, because `up` reuses whatever `latest` the host already has.
|
||||
|
||||
**A build that fails with `429 Too Many Requests` on a base image** is Docker Hub rate-limiting
|
||||
this host. There is no Docker Hub login here, and the build asks about `debian:bookworm-slim` and
|
||||
`rust:1-slim-bookworm` every time unless they are already stored locally. Pull them from Google's
|
||||
mirror and tag them; the build then uses the local copies without asking Docker Hub:
|
||||
|
||||
```sh
|
||||
docker pull mirror.gcr.io/library/debian:bookworm-slim
|
||||
docker tag mirror.gcr.io/library/debian:bookworm-slim debian:bookworm-slim
|
||||
docker pull mirror.gcr.io/library/rust:1-slim-bookworm
|
||||
docker tag mirror.gcr.io/library/rust:1-slim-bookworm rust:1-slim-bookworm
|
||||
```
|
||||
|
||||
Run those again now and then, or the local copies go stale.
|
||||
|
||||
The healthcheck runs `ipx status` against the control socket, so `(healthy)` in `docker ps` means
|
||||
the daemon answers there and can read its database, not just that the web port is up. The socket
|
||||
answers `status` itself instead of queuing it behind the worker's current job, so a long scan or
|
||||
download does not fail the check; it also means a worker stuck on one job would still pass. The
|
||||
container restarts on its own after a reboot.
|
||||
|
||||
Before the container, ipx ran by hand in code-server, with its files in `/config/.config/ipx/` and
|
||||
`/config/.local/share/ipx/`. Those are still there and the container does not read them. If you run
|
||||
a daemon by hand for testing, stop it with **`pkill -x ipx`, never `pkill -f ipx`**. `-f` matches
|
||||
the shell running the command and kills the session (exit 144). This has happened more than once.
|
||||
|
||||
## Before you touch the page
|
||||
|
||||
`web/index.html` is `include_str!`d into the binary, so **every page change needs a rebuild** before
|
||||
it is visible. It is one file: markup, CSS and script.
|
||||
|
||||
After any edit to it:
|
||||
|
||||
```sh
|
||||
node tests/page-smoke.js
|
||||
```
|
||||
|
||||
That loads the script against a stub DOM and checks every selector it wires at load actually
|
||||
exists. It exists because a patch once anchored on a deleted function, `String.replace` silently
|
||||
matched nothing, and the whole UI died with a `ReferenceError` while every server-side test passed.
|
||||
|
||||
Patching that file by guessing an anchor string has failed repeatedly. Read the exact block first
|
||||
(`sed -n 'START,ENDp'`), match it verbatim, and assert the replacement happened rather than hoping.
|
||||
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
cargo test # ~51 tests: parsing, filters, retention, schedules, SQL, per-user state
|
||||
node tests/page-smoke.js
|
||||
npx playwright test # 16 browser tests against a real daemon on fixture feeds
|
||||
```
|
||||
|
||||
Things about the browser suite that have cost time:
|
||||
|
||||
* It starts **its own daemon and database** under `/tmp/ipx-ui-test`, wiped once per run. Playwright
|
||||
re-imports the config in every worker, so `prepare()` guards on `TEST_WORKER_INDEX` — without
|
||||
that guard a worker deleted the database out from under the running daemon, which then kept
|
||||
serving from the unlinked inode while everything else saw an empty file.
|
||||
* Tests **share that daemon and run in order**. A test that opens an item marks it read and changes
|
||||
what later tests see. Write assertions that do not depend on what ran before, or normalise the
|
||||
state first.
|
||||
* Fixture feeds must not share an enclosure URL, because `enclosures.url` is globally unique and
|
||||
whichever feed is scanned first claims it.
|
||||
* `webServer` starts **before** `globalSetup`, which is why the fixture config is written at
|
||||
config-load time instead.
|
||||
|
||||
Non-trivial logic leaves one runnable check behind. Pure functions (`merge_policy`, `pick`,
|
||||
`matches_keywords`, `parse_interval`) are the easiest place to put it.
|
||||
|
||||
## Things that are true and easy to get wrong
|
||||
|
||||
* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
|
||||
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
|
||||
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
|
||||
and `position` columns from before accounts; two bugs came from queries still reading them
|
||||
(retention, and the entry pruner), and `migrate()` now drops them.
|
||||
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
|
||||
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
|
||||
derived and never written to config.
|
||||
* **One fetch serves everyone**, so scan policy is a union of subscribers' wants (`merge_policy`).
|
||||
Anyone wanting an item is enough to fetch it.
|
||||
* **The UI hiding a control is not enforcement.** Admin-only actions check `user.is_admin` in the
|
||||
handler and return `403`.
|
||||
* **A `tokio::select!` only races its branches at the point of selection.** A long download has to
|
||||
watch the shutdown channel itself; the daemon ignored SIGTERM for exactly this reason.
|
||||
* Only one daemon per socket. Removing the socket file defeats the guard and you get two daemons
|
||||
fighting over the database, with the stale one still holding the port.
|
||||
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
|
||||
different task. `ipx status` checks the control socket and the database; to see the worker
|
||||
getting through its jobs, watch for `scan complete` in the log.
|
||||
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's
|
||||
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on
|
||||
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait
|
||||
for `daemon started` in the log. Copy `state.db` aside before deploying one.
|
||||
|
||||
## House style
|
||||
|
||||
Comments explain **why**, not what. If a line looks odd, the comment says what went wrong without
|
||||
it. No emoji, no exclamation marks, no "obviously". Prose in the UI and docs is plain English and
|
||||
addressed to the person using it.
|
||||
|
||||
Every change gets one line under `## [Unreleased]` in [CHANGELOG.md](CHANGELOG.md), in its
|
||||
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) group: Added, Changed, Deprecated,
|
||||
Removed, Fixed or Security. Say it the way someone using ipx would notice it. When there is more to
|
||||
say, such as what was wrong before or what it cost to find out, write it up at the top of
|
||||
[docs/history.md](docs/history.md), dated. That record has been more useful than the git log more
|
||||
than once.
|
||||
|
||||
Cutting a release: rename `[Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD` and open a new empty
|
||||
`[Unreleased]` above it, bump `version` in `Cargo.toml`, tag the commit `vX.Y.Z`, and update the
|
||||
compare links at the bottom of the changelog.
|
||||
|
||||
Deliberate simplifications get a `ponytail:` comment naming the ceiling and the upgrade path, e.g.
|
||||
`// ponytail: global connection mutex, move to a pool if feed count makes it contend`.
|
||||
|
||||
## Known gaps
|
||||
|
||||
* Cloudflare's `Cf-Access-Jwt-Assertion` is not verified — ipx trusts the hop plus `trusted_proxies`
|
||||
(documented in [docs/sso.md](docs/sso.md)).
|
||||
* A feed's `<description>` subtitle is dropped whenever `content:encoded` exists, which loses
|
||||
Substack-style subtitles.
|
||||
361
Cargo.lock
generated
361
Cargo.lock
generated
@@ -23,6 +23,18 @@ version = "0.2.21"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
|
||||
|
||||
[[package]]
|
||||
name = "ammonia"
|
||||
version = "4.1.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "dc6d763210e2eb7670d1a5183a08bebefa3f97db2a738a684f2ce00bd49f681d"
|
||||
dependencies = [
|
||||
"cssparser",
|
||||
"html5ever",
|
||||
"maplit",
|
||||
"url",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "android_system_properties"
|
||||
version = "0.1.6"
|
||||
@@ -97,6 +109,18 @@ dependencies = [
|
||||
"rustversion",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "argon2"
|
||||
version = "0.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "134c52ddac6d63c576bef8168db10c83c49c26444ecbc68060fef078925a901c"
|
||||
dependencies = [
|
||||
"base64ct",
|
||||
"blake2",
|
||||
"cpufeatures",
|
||||
"password-hash",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "arrayvec"
|
||||
version = "0.7.8"
|
||||
@@ -175,7 +199,7 @@ dependencies = [
|
||||
"chrono",
|
||||
"derive_builder",
|
||||
"diligent-date-parser",
|
||||
"quick-xml",
|
||||
"quick-xml 0.41.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -222,6 +246,7 @@ checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90"
|
||||
dependencies = [
|
||||
"axum-core",
|
||||
"bytes",
|
||||
"form_urlencoded",
|
||||
"futures-util",
|
||||
"http",
|
||||
"http-body",
|
||||
@@ -235,6 +260,9 @@ dependencies = [
|
||||
"percent-encoding",
|
||||
"pin-project-lite",
|
||||
"serde_core",
|
||||
"serde_json",
|
||||
"serde_path_to_error",
|
||||
"serde_urlencoded",
|
||||
"sync_wrapper",
|
||||
"tokio",
|
||||
"tower",
|
||||
@@ -310,6 +338,12 @@ version = "0.23.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
|
||||
|
||||
[[package]]
|
||||
name = "base64ct"
|
||||
version = "1.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06"
|
||||
|
||||
[[package]]
|
||||
name = "bitflags"
|
||||
version = "1.3.2"
|
||||
@@ -334,6 +368,24 @@ dependencies = [
|
||||
"wyz",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "blake2"
|
||||
version = "0.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5b5d4d889834ee8ecfc0f8426ad30faf7cdcb10f741a8e6d7224d95325479f6f"
|
||||
dependencies = [
|
||||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "block-buffer"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa"
|
||||
dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "bs58"
|
||||
version = "0.5.1"
|
||||
@@ -384,17 +436,6 @@ dependencies = [
|
||||
"shlex",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfb"
|
||||
version = "0.14.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a347dcabdae9c31b0825fd6a8bed285ec9c2acb89c47827126d52fa4f59cece3"
|
||||
dependencies = [
|
||||
"fnv",
|
||||
"uuid",
|
||||
"web-time",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfg-if"
|
||||
version = "1.0.4"
|
||||
@@ -492,6 +533,12 @@ dependencies = [
|
||||
"cc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cmov"
|
||||
version = "0.5.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a"
|
||||
|
||||
[[package]]
|
||||
name = "colorchoice"
|
||||
version = "1.0.5"
|
||||
@@ -590,6 +637,35 @@ version = "0.8.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a31eee39dddec8330830986fcd7625edb5a24ec90ea038215273bbc3adb08ac6"
|
||||
|
||||
[[package]]
|
||||
name = "crypto-common"
|
||||
version = "0.2.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"
|
||||
dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cssparser"
|
||||
version = "0.37.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8c9cdaae01d5ed7882b04d795e7f752f46ff52d2fa3b50a20d28c464510bba98"
|
||||
dependencies = [
|
||||
"dtoa-short",
|
||||
"itoa",
|
||||
"smallvec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ctutils"
|
||||
version = "0.4.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e"
|
||||
dependencies = [
|
||||
"cmov",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "darling"
|
||||
version = "0.20.11"
|
||||
@@ -751,6 +827,17 @@ dependencies = [
|
||||
"syn 2.0.119",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "digest"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
|
||||
dependencies = [
|
||||
"block-buffer",
|
||||
"crypto-common",
|
||||
"ctutils",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "diligent-date-parser"
|
||||
version = "0.1.5"
|
||||
@@ -769,15 +856,6 @@ dependencies = [
|
||||
"dirs-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dirs"
|
||||
version = "7.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8d57d423b3c82e89b9a24ca3091fee61f456a26edbd28d26c65906f4bc1dcd8f"
|
||||
dependencies = [
|
||||
"dirs-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dirs-sys"
|
||||
version = "0.5.0"
|
||||
@@ -813,6 +891,21 @@ dependencies = [
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dtoa"
|
||||
version = "1.0.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4c3cf4824e2d5f025c7b531afcb2325364084a16806f6d47fbc1f5fbd9960590"
|
||||
|
||||
[[package]]
|
||||
name = "dtoa-short"
|
||||
version = "0.3.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cd1511a7b6a56299bd043a9c167a6d2bfb37bf84a6dfceaba651168adfb43c87"
|
||||
dependencies = [
|
||||
"dtoa",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dunce"
|
||||
version = "1.0.5"
|
||||
@@ -1206,6 +1299,16 @@ version = "0.4.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
|
||||
|
||||
[[package]]
|
||||
name = "html5ever"
|
||||
version = "0.39.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "46a1761807faccc9a19e86944bbf40610014066306f96edcdedc2fb714bcb7b8"
|
||||
dependencies = [
|
||||
"log",
|
||||
"markup5ever",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "http"
|
||||
version = "1.5.0"
|
||||
@@ -1239,6 +1342,12 @@ dependencies = [
|
||||
"pin-project-lite",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "http-range-header"
|
||||
version = "0.4.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9171a2ea8a68358193d15dd5d70c1c10a2afc3e7e4c5bc92bc9f025cebd7359c"
|
||||
|
||||
[[package]]
|
||||
name = "httparse"
|
||||
version = "1.10.1"
|
||||
@@ -1251,6 +1360,15 @@ version = "1.0.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
|
||||
|
||||
[[package]]
|
||||
name = "hybrid-array"
|
||||
version = "0.4.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "27f864f10dfb56725ce5ce5472bc52252c8f93a4ab86327122cebf62c5f59a17"
|
||||
dependencies = [
|
||||
"typenum",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hyper"
|
||||
version = "1.11.1"
|
||||
@@ -1470,15 +1588,6 @@ dependencies = [
|
||||
"serde_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "infer"
|
||||
version = "0.22.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f4200d433cbd5178df7797c9c2e75b348b728e39631cf14520d1e2fc424201f4"
|
||||
dependencies = [
|
||||
"cfb",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "intervaltree"
|
||||
version = "0.2.7"
|
||||
@@ -1496,18 +1605,20 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
|
||||
|
||||
[[package]]
|
||||
name = "ipx"
|
||||
version = "0.1.0"
|
||||
version = "0.5.3"
|
||||
dependencies = [
|
||||
"ammonia",
|
||||
"anyhow",
|
||||
"argon2",
|
||||
"atom_syndication",
|
||||
"axum",
|
||||
"chrono",
|
||||
"clap",
|
||||
"dirs",
|
||||
"futures-util",
|
||||
"infer",
|
||||
"librqbit",
|
||||
"opml",
|
||||
"percent-encoding",
|
||||
"quick-xml 0.42.0",
|
||||
"reqwest",
|
||||
"rss",
|
||||
"rusqlite",
|
||||
@@ -1515,6 +1626,8 @@ dependencies = [
|
||||
"serde_json",
|
||||
"tokio",
|
||||
"toml",
|
||||
"tower",
|
||||
"tower-http 0.7.1",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
"url",
|
||||
@@ -1974,7 +2087,7 @@ dependencies = [
|
||||
"httparse",
|
||||
"librqbit-dualstack-sockets",
|
||||
"network-interface",
|
||||
"quick-xml",
|
||||
"quick-xml 0.41.0",
|
||||
"reqwest",
|
||||
"serde",
|
||||
"serde_derive",
|
||||
@@ -2045,6 +2158,23 @@ version = "0.1.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
|
||||
|
||||
[[package]]
|
||||
name = "maplit"
|
||||
version = "1.0.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3e2e65a1a2e43cfcb47a895c4c8b10d1f4a61097f9f254f183aee60cad9c651d"
|
||||
|
||||
[[package]]
|
||||
name = "markup5ever"
|
||||
version = "0.39.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7122d987ec5f704ee56f6e5b41a7d93722e9aae27ae07cafa4036c4d3f9757de"
|
||||
dependencies = [
|
||||
"log",
|
||||
"tendril",
|
||||
"web_atoms",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "matchers"
|
||||
version = "0.2.0"
|
||||
@@ -2161,6 +2291,12 @@ dependencies = [
|
||||
"winapi",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "new_debug_unreachable"
|
||||
version = "1.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "650eef8c711430f1a879fdd01d4745a7deea475becfb90269c06775983bbf086"
|
||||
|
||||
[[package]]
|
||||
name = "nix"
|
||||
version = "0.30.1"
|
||||
@@ -2314,12 +2450,72 @@ dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "password-hash"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "aab41826031698d6ffcd9cff78ef56ef998e39dc7e5067cdfebe373842d4723b"
|
||||
dependencies = [
|
||||
"getrandom 0.4.3",
|
||||
"phc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "percent-encoding"
|
||||
version = "2.3.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
|
||||
|
||||
[[package]]
|
||||
name = "phc"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "44dc769b75f93afdddd8c7fa12d685292ddeff1e66f7f0f3a234cf1818afe892"
|
||||
dependencies = [
|
||||
"base64ct",
|
||||
"ctutils",
|
||||
"getrandom 0.4.3",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf"
|
||||
version = "0.13.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c1562dc717473dbaa4c1f85a36410e03c047b2e7df7f45ee938fbef64ae7fadf"
|
||||
dependencies = [
|
||||
"phf_shared",
|
||||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_codegen"
|
||||
version = "0.13.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "49aa7f9d80421bca176ca8dbfebe668cc7a2684708594ec9f3c0db0805d5d6e1"
|
||||
dependencies = [
|
||||
"phf_generator",
|
||||
"phf_shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_generator"
|
||||
version = "0.13.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "135ace3a761e564ec88c03a77317a7c6b80bb7f7135ef2544dbe054243b89737"
|
||||
dependencies = [
|
||||
"fastrand",
|
||||
"phf_shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_shared"
|
||||
version = "0.13.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e57fef6bc5981e38c2ce2d63bfa546861309f875b8a75f092d1d54ae2d64f266"
|
||||
dependencies = [
|
||||
"siphasher",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pin-project-lite"
|
||||
version = "0.2.17"
|
||||
@@ -2371,6 +2567,12 @@ dependencies = [
|
||||
"zerocopy",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "precomputed-hash"
|
||||
version = "0.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "925383efa346730478fb4838dbe9137d2a47675ad789c546d150a6e1dd4ab31c"
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.107"
|
||||
@@ -2406,6 +2608,15 @@ dependencies = [
|
||||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quick-xml"
|
||||
version = "0.42.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "41b1177fdf999d2321d3fb46ff47159d9c1fb9ad66a4879f8c50a0b504615e9b"
|
||||
dependencies = [
|
||||
"memchr",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quinn"
|
||||
version = "0.11.11"
|
||||
@@ -2666,7 +2877,7 @@ dependencies = [
|
||||
"tokio-rustls",
|
||||
"tokio-util",
|
||||
"tower",
|
||||
"tower-http",
|
||||
"tower-http 0.6.11",
|
||||
"tower-service",
|
||||
"url",
|
||||
"wasm-bindgen",
|
||||
@@ -2727,7 +2938,7 @@ checksum = "f505d3e5e7b06b4dc0245b13294f8ef9a1a0f70284708be1e11c5b7b7441034e"
|
||||
dependencies = [
|
||||
"atom_syndication",
|
||||
"derive_builder",
|
||||
"quick-xml",
|
||||
"quick-xml 0.41.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3092,6 +3303,12 @@ version = "0.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
|
||||
|
||||
[[package]]
|
||||
name = "siphasher"
|
||||
version = "1.0.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
|
||||
|
||||
[[package]]
|
||||
name = "size_format"
|
||||
version = "1.0.2"
|
||||
@@ -3151,6 +3368,30 @@ version = "1.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
|
||||
|
||||
[[package]]
|
||||
name = "string_cache"
|
||||
version = "0.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a18596f8c785a729f2819c0f6a7eae6ebeebdfffbfe4214ae6b087f690e31901"
|
||||
dependencies = [
|
||||
"new_debug_unreachable",
|
||||
"parking_lot",
|
||||
"phf_shared",
|
||||
"precomputed-hash",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "string_cache_codegen"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "585635e46db231059f76c5849798146164652513eb9e8ab2685939dd90f29b69"
|
||||
dependencies = [
|
||||
"phf_generator",
|
||||
"phf_shared",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "strsim"
|
||||
version = "0.11.1"
|
||||
@@ -3243,6 +3484,15 @@ version = "1.0.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369"
|
||||
|
||||
[[package]]
|
||||
name = "tendril"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5fed54709c5b3a53d09bb1c113ea4f5ceafd1e772ddcb0030a82e1d56c087b08"
|
||||
dependencies = [
|
||||
"new_debug_unreachable",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "thiserror"
|
||||
version = "1.0.69"
|
||||
@@ -3474,6 +3724,7 @@ dependencies = [
|
||||
"tokio",
|
||||
"tower-layer",
|
||||
"tower-service",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3499,6 +3750,31 @@ dependencies = [
|
||||
"url",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tower-http"
|
||||
version = "0.7.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "08a05a66a4fdd61cbbe0a1d755ffe0ca6aba159dd4820936a0ff8a8278245b9c"
|
||||
dependencies = [
|
||||
"bitflags 2.13.1",
|
||||
"bytes",
|
||||
"futures-core",
|
||||
"futures-util",
|
||||
"http",
|
||||
"http-body",
|
||||
"http-body-util",
|
||||
"http-range-header",
|
||||
"httpdate",
|
||||
"mime",
|
||||
"mime_guess",
|
||||
"percent-encoding",
|
||||
"pin-project-lite",
|
||||
"tokio",
|
||||
"tokio-util",
|
||||
"tower-layer",
|
||||
"tower-service",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tower-layer"
|
||||
version = "0.3.3"
|
||||
@@ -3517,6 +3793,7 @@ version = "0.1.44"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
|
||||
dependencies = [
|
||||
"log",
|
||||
"pin-project-lite",
|
||||
"tracing-attributes",
|
||||
"tracing-core",
|
||||
@@ -3790,6 +4067,18 @@ dependencies = [
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "web_atoms"
|
||||
version = "0.2.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ba8b815c1b593dc0baf78dd0f4fc8fdb2de53198fb1163738093e9a311c33fb3"
|
||||
dependencies = [
|
||||
"phf",
|
||||
"phf_codegen",
|
||||
"string_cache",
|
||||
"string_cache_codegen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "webpki-root-certs"
|
||||
version = "1.0.9"
|
||||
|
||||
10
Cargo.toml
10
Cargo.toml
@@ -1,19 +1,21 @@
|
||||
[package]
|
||||
name = "ipx"
|
||||
version = "0.1.0"
|
||||
version = "0.5.3"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
ammonia = "4.1.4"
|
||||
anyhow = "1.0.104"
|
||||
argon2 = "0.6.0"
|
||||
atom_syndication = "0.12.10"
|
||||
axum = "0.8.9"
|
||||
chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] }
|
||||
clap = { version = "4.6.6", features = ["derive"] }
|
||||
dirs = "7.0.0"
|
||||
futures-util = { version = "0.3.34", default-features = false, features = ["std"] }
|
||||
infer = "0.22.0"
|
||||
librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] }
|
||||
opml = "1.1.6"
|
||||
percent-encoding = "2.3.2"
|
||||
quick-xml = "0.42.0"
|
||||
reqwest = { version = "0.13.5", default-features = false, features = ["rustls", "http2", "gzip", "stream", "json", "charset", "system-proxy"] }
|
||||
rss = "2.1.1"
|
||||
rusqlite = { version = "0.40.2", features = ["bundled"] }
|
||||
@@ -21,6 +23,8 @@ serde = { version = "1.0.229", features = ["derive"] }
|
||||
serde_json = "1.0.151"
|
||||
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
|
||||
toml = "1.1.5"
|
||||
tower = { version = "0.5.3", features = ["util"] }
|
||||
tower-http = { version = "0.7.1", features = ["fs"] }
|
||||
tracing = "0.1.44"
|
||||
tracing-subscriber = { version = "0.3.23", features = ["env-filter"] }
|
||||
url = "2.5.8"
|
||||
|
||||
39
Dockerfile
Normal file
39
Dockerfile
Normal file
@@ -0,0 +1,39 @@
|
||||
# Build. rusqlite is bundled (compiles SQLite from source) and librqbit needs a C
|
||||
# toolchain, so the builder needs cc. TLS is rustls throughout, so no OpenSSL headers.
|
||||
FROM rust:1-slim-bookworm AS build
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /src
|
||||
|
||||
# Dependencies first, so editing the source does not rebuild librqbit every time.
|
||||
COPY Cargo.toml Cargo.lock ./
|
||||
RUN mkdir src && echo 'fn main(){}' > src/main.rs \
|
||||
&& cargo build --release --locked \
|
||||
&& rm -rf src
|
||||
|
||||
COPY src ./src
|
||||
COPY web ./web
|
||||
# cargo skips a rebuild if mtimes look untouched; make sure it does not.
|
||||
RUN touch src/main.rs && cargo build --release --locked
|
||||
|
||||
FROM debian:bookworm-slim
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates gosu \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
COPY --from=build /src/target/release/ipx /usr/local/bin/ipx
|
||||
COPY docker-entrypoint.sh /usr/local/bin/
|
||||
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
|
||||
|
||||
ENV IPX_CONFIG=/config/config.toml \
|
||||
IPX_DATA_DIR=/data \
|
||||
IPX_LOG=ipx=info \
|
||||
PUID=99 \
|
||||
PGID=100
|
||||
VOLUME ["/config", "/data", "/downloads"]
|
||||
|
||||
# Web UI, and the BitTorrent peer port (TCP and UDP -- DHT needs the UDP side).
|
||||
EXPOSE 8099/tcp 6881/tcp 6881/udp
|
||||
|
||||
ENTRYPOINT ["docker-entrypoint.sh"]
|
||||
CMD ["ipx", "daemon"]
|
||||
262
PROGRESS.md
262
PROGRESS.md
@@ -1,262 +0,0 @@
|
||||
# Progress
|
||||
|
||||
Running record of what has actually landed. Newest entry first.
|
||||
The full design and step list live in the plan file at
|
||||
`/config/.claude/plans/i-want-to-create-playful-quiche.md`.
|
||||
|
||||
## Build order
|
||||
|
||||
- [x] **1. Repo skeleton** — git init (`main`), `cargo init --name ipx`, deps pinned, LICENSE,
|
||||
README, this file.
|
||||
- [x] **2. `config.rs` + `db.rs`** — TOML config structs + SQLite schema.
|
||||
- [x] **3. `feed.rs`** — conditional GET, RSS-then-Atom parse, persist entries.
|
||||
- [x] **4. `download.rs`** — downloads, filters, dedupe.
|
||||
- [x] **5. `retention.rs`** — oldest-first quota + age reaper.
|
||||
- [x] **6. `ipc.rs` + daemon** — UDS JSON-lines server, TTL scheduler, CLI-proxies-to-daemon.
|
||||
- [x] **7. `torrent.rs`** — librqbit, seed to ratio/time, stall abort. (swarm download unverified —
|
||||
see the step 7 entry)
|
||||
- [x] **8. OPML + polish** — import/export, add/rm/status, tracing setup, systemd units, README.
|
||||
|
||||
## Smoke tests
|
||||
|
||||
1. `ipx add <feed>` + `ipx fetch` → file in `download_dir/<Show>/`, row in `enclosures`.
|
||||
2. `ipx fetch` again → no re-download, feed skipped for TTL.
|
||||
3. `ipx daemon &` + `nc -U $XDG_RUNTIME_DIR/ipx.sock`, send `{"cmd":"fetch"}` → JSON events;
|
||||
a concurrent `ipx fetch` proxies to the daemon instead of downloading in parallel.
|
||||
4. Delete a downloaded file by hand, `ipx fetch` → NOT re-downloaded.
|
||||
5. Torrent enclosure → downloads, moves, stops seeding at the configured ratio/time.
|
||||
6. `ipx reap --dry-run` under quota pressure → oldest-first hit list; real run flips rows to
|
||||
`reaped`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-09 — Step 8: OPML and polish
|
||||
|
||||
`ipx add <url>` fetches the feed to name it from its own title (`Accidental Tech Podcast` ->
|
||||
`accidental-tech-podcast`); a feed that cannot be reached is still added, named from its URL, rather
|
||||
than refused. `ipx rm` leaves downloads and history alone, so re-adding a feed does not re-pull its
|
||||
back catalogue. `ipx import`/`export` walk nested OPML folder outlines and skip URLs already
|
||||
subscribed. `tracing` logs to stderr, `IPX_LOG` sets the filter. `contrib/` has a systemd user unit
|
||||
for the daemon, plus a timer and one-shot service for the no-daemon style (with the caveat that
|
||||
without a daemon there is no socket for a UI).
|
||||
|
||||
Verified: `cargo test` 27/27. Round-trip — exported 3 feeds with real titles, removed one,
|
||||
re-imported: exactly the missing one came back, no duplicates, config still mode 0600. Quickstart
|
||||
from a genuinely empty home with no env overrides: `list` on a missing config, `add`, `fetch`
|
||||
(1 downloaded, 1 explicit skipped, 1 torrent 404), `list`; every path resolved under `$HOME`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-09 — Step 7: torrent.rs
|
||||
|
||||
`src/torrent.rs`: a librqbit `Session` started lazily on first torrent (binding ports and starting a
|
||||
DHT for a config that has never seen a torrent would be rude). `.torrent` URLs and magnets both go
|
||||
through `AddTorrent::from_url`. Progress is polled once a second and reported through the same
|
||||
`Progress` events HTTP downloads use.
|
||||
|
||||
Downloads go **straight into the feed folder** rather than staging and moving. The plan said move
|
||||
then seed, which cannot work — seeding serves the files it downloaded, so moving them first breaks
|
||||
it. In-place also removes a copy the original had to do.
|
||||
|
||||
Seeding stops at `seed_ratio` or `seed_time_mins`, whichever comes first, then the torrent is
|
||||
released from the session (files kept).
|
||||
|
||||
**Bug the smoke test caught:** the stall budget only covered the download loop, but resolving a
|
||||
magnet's metadata happens inside `add_torrent`, which against a dead swarm never returns — a
|
||||
torrent nobody seeds wedged the scan indefinitely. `add_torrent` is now wrapped in the same budget.
|
||||
|
||||
Verified: `cargo test` 27/27 (ratio incl. the divide-by-zero case, stall_mins = 0 not meaning
|
||||
"abort instantly"). Routing: with `enabled = false` a torrent enclosure is marked
|
||||
`skipped/torrents disabled` and never attempted. Session startup works here. Stall abort measured
|
||||
end to end: with `stall_mins = 1`, a dead magnet failed at 20:58:23 -> 20:59:24, exactly 61s, and
|
||||
the row recorded `error / no metadata after 1 minutes, gave up`.
|
||||
|
||||
**Not verified: an actual successful swarm download.** This sandbox has no reachable peers, so
|
||||
smoke 5's happy path — payload lands, seeding stops at the ratio — has not been run. The code paths
|
||||
either side of it are tested; the swarm itself needs a real network. Worth running once against a
|
||||
live torrent feed before trusting it.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-09 — Step 6: ipc.rs + daemon
|
||||
|
||||
`src/ipc.rs`: `Event` and `Command` as serde-tagged enums (`{"ev":...}` / `{"cmd":...}`), one JSON
|
||||
object per line over a Unix socket. `Emitter` is the single output path — it broadcasts to socket
|
||||
clients, prints the human rendering to a terminal, or both, so the scan code no longer knows how it
|
||||
is being watched. That replaces `printMSG` and its `;;1;;1;;100.00;;42.31` sentinels.
|
||||
|
||||
`main.rs` restructured around a `Ctx` (config, db, client, emitter). `ipx daemon` binds the socket,
|
||||
serves clients, and runs a 60s ticker that defers to each feed's TTL. Commands from every client
|
||||
funnel through one mpsc queue into a single worker, which is what stops two scans overlapping.
|
||||
Any CLI subcommand with a wire form probes the socket first and proxies to a running daemon;
|
||||
`--local` forces the work to happen in-process. SIGINT/SIGTERM remove the socket on the way out.
|
||||
|
||||
**Bug the smoke test caught:** `fetch` runs a retention sweep first, and that sweep was emitting the
|
||||
terminal `ReapDone`. A UI waiting for its fetch to finish would have stopped reading before the scan
|
||||
started. Only a standalone `ipx reap` emits it now.
|
||||
|
||||
Verified: `cargo test` 24/24. Smoke 3 in full — daemon starts and binds; a raw socket client sending
|
||||
`{"cmd":"fetch","force":true}` gets `feed_start` → 14 throttled `progress` events → `download_done`
|
||||
→ `feed_done` → `scan_done`; `{"cmd":"status"}` answers `{"ev":"status","feeds":1,...}`. With the
|
||||
daemon up, `ipx fetch` from the CLI logged `command from a client cmd=Fetch { .. }` in the daemon
|
||||
and rendered the streamed events, so it proxied rather than downloading in parallel. Socket removed
|
||||
on SIGTERM; with no daemon the same command runs locally.
|
||||
|
||||
Deferred: `{"cmd":"cancel","enclosure":N}` from the plan's protocol is **not implemented** —
|
||||
downloads run sequentially in one worker, so there is nothing to cancel concurrently yet. It wants
|
||||
a per-download cancellation token, which is worth doing when downloads go parallel. Say if you want
|
||||
it sooner.
|
||||
|
||||
Note: the event stream is a broadcast, so a CLI client seeing a busy daemon also sees that other
|
||||
work. Fine for a terminal; a UI wanting strict request/response would want per-request ids.
|
||||
|
||||
Also note: the daemon reads config once at startup — changing `config.toml` needs a restart.
|
||||
|
||||
Next: step 7 — `torrent.rs`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-09 — Step 5: retention.rs
|
||||
|
||||
`src/retention.rs`: reconcile pass (rows claiming a file that is gone become `reaped`, fixing the
|
||||
step-4 wart), age sweep, quota sweep keeping the original's 50 MB headroom pad, and entry pruning.
|
||||
`ipx reap [--dry-run]`; a sweep also runs before every `fetch`, as the Python did per download.
|
||||
`pick()` and `aged()` are pure so the ordering rules are testable without touching a disk.
|
||||
|
||||
**Judgement call worth Ray's eye.** The Python meant to reap only `read = 1 AND flagged = 0` but
|
||||
never managed it — a missing `plistlib` import and an `EntreiesData` typo made that filter throw on
|
||||
every candidate, so with a `.ipxd` present nothing was ever deleted. Requiring `read = 1` here would
|
||||
be equally dead, because nothing marks episodes read until a UI exists. So: **`flagged` is the
|
||||
keep-forever marker, and `read` only decides what goes first** (`ORDER BY read DESC, downloaded_at
|
||||
ASC`). Quota therefore actually reclaims space headless. Say the word if you would rather unread
|
||||
episodes were never touched.
|
||||
|
||||
Second call: `max_age_days` deletes *files* older than the cutoff, not just fileless entries as the
|
||||
plan's wording had it — "keep 30 days of episodes" is what the setting reads like on a NAS.
|
||||
|
||||
Verified: `cargo test` 21/21, including the two tests encoding the exact bug the Python had —
|
||||
flagged files are never offered, and read sort ahead of unread. Smoke 6 with three 30 MB episodes
|
||||
against a 0.1 GB quota (52.4 MB ceiling after the pad): dry run listed ep1+ep2 and deleted nothing
|
||||
(3 files still on disk), the real run deleted exactly those two oldest, left ep3, flipped both rows
|
||||
to `reaped` with `path = NULL`. A full re-parse with the conditional-GET headers cleared then
|
||||
re-downloaded nothing.
|
||||
|
||||
Next: step 6 — `ipc.rs` + daemon.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-09 — Step 4: download.rs
|
||||
|
||||
`src/download.rs`: streaming download to `<download_dir>/.ipx-incomplete/` (same filesystem as the
|
||||
destination, so filing it is a rename, not the original's copy-then-unlink), content sniffing,
|
||||
then `place()`. Filename comes from the URL's last path segment, percent-decoded, unless
|
||||
`Content-Disposition` names one (RFC 5987 `filename*=` preferred). The sanitizer keeps UTF-8 —
|
||||
`latin1_to_ascii` existed because 2004 filesystems demanded ASCII — strips the same characters
|
||||
`stringCleaning()` did plus control chars, and adds a real 255-byte cap the Python never had,
|
||||
preserving the extension across truncation.
|
||||
|
||||
Sniffing replaces `detectFileType()`, which called a `typeFile` module that was already missing in
|
||||
2008 and so always answered `'data'`. Two rules survive: an HTML body is a failed download (login
|
||||
wall/error page), and a torrent body is a torrent whatever the MIME claimed.
|
||||
|
||||
Filters run once at discovery and are recorded in `enclosures.state`; the download queue is then
|
||||
just "everything still `pending`", so an enclosure held back by `max_new_per_check` is picked up by
|
||||
the next scan instead of being lost. Keywords are OR'd across keywords and AND'd within one — the
|
||||
original's nested loop let a later keyword silently undo an earlier miss.
|
||||
|
||||
Verified: `cargo test` 16/16. Smoke against a local server, five enclosures, each filter path hit:
|
||||
`ep1.mp3 -> done`, `ep2.mp3 -> skipped (explicit)`, `ep2.mp3?v=3 -> skipped (no keyword match)`,
|
||||
`paywall.html -> error (HTML page, not media)`, `ep5.torrent -> torrent (deferred to step 7)`.
|
||||
Smoke 2 and 4 pass, and because a plain rerun 304s before parsing, dedupe was proved separately by
|
||||
clearing the stored etag/last-modified and re-parsing all five entries: 0 downloaded, hand-deleted
|
||||
file not refetched, `.ipx-incomplete` left empty.
|
||||
|
||||
Known wart: `ipx list` counts `path IS NOT NULL`, so a hand-deleted file still reads as downloaded.
|
||||
Reconciling rows against the filesystem belongs in step 5.
|
||||
Next: step 5 — `retention.rs`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-09 — Step 3: feed.rs
|
||||
|
||||
`src/feed.rs`: conditional GET (`If-None-Match` + `If-Modified-Since`, optional basic auth) and a
|
||||
RSS-first / Atom-fallback parser normalising both into `ParsedFeed`/`Entry`/`Enclosure`. Feed-level
|
||||
`itunes:explicit` overrides the entry level, as the original did. `<ttl>` is captured. RSS
|
||||
`content:encoded` wins over `description`. Atom enclosures come only from `rel="enclosure"` links.
|
||||
|
||||
GUID: the original hashed the title or description when no guid existed; here the chain is
|
||||
guid → permalink → enclosure URL → title, all stable identifiers, so no hashing and no MD5
|
||||
dependency. An entry with none of them has nothing to download and is dropped.
|
||||
|
||||
`db.rs` gained `http_state`, `record_feed`, `touch_feed`, `set_feed_error`, `record_entry`,
|
||||
`record_enclosure`. A changed title/description flips `read` back to 0 — what the original's
|
||||
textDiff was ultimately for, minus the `<ins>`/`<del>` markup, which belongs in the UI.
|
||||
|
||||
`main.rs` gained `ipx fetch [FEED] [--force]`. A failing feed records its error and the scan
|
||||
continues.
|
||||
|
||||
Verified: `cargo test` 10/10. Gate met against a local `python3 -m http.server` serving the
|
||||
fixtures — first run inserted 4 entries + 4 enclosures across an RSS and an Atom feed; second run
|
||||
showed both skip paths, `atomcast: not modified` (304) and `testcast: not due for 45m` (the feed's
|
||||
own ttl=45 beating `interval_mins = 0`).
|
||||
|
||||
Deferred: nothing downloads yet — enclosure rows land in state `pending`. That is step 4.
|
||||
Next: step 4 — `download.rs`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-09 — Step 2: config.rs + db.rs
|
||||
|
||||
`src/config.rs`: serde structs for `[general]`, `[torrent]` and `[feeds.<id>]` with defaults, `~`
|
||||
expansion, `IPX_CONFIG` / `IPX_DATA_DIR` overrides, `save()` at mode 0600, `Feed::password()`
|
||||
(`password_env` beats a literal `password`), `Torrent::ports()` parsing `"6881-6889"`. A missing
|
||||
config file loads as an empty one so a fresh install works. Retention defaults are 0/0
|
||||
(unlimited, keep forever) — nothing gets deleted until Ray asks for it.
|
||||
|
||||
`src/db.rs`: schema exactly as planned, WAL + `busy_timeout`, `Db::open()` idempotent, connection
|
||||
behind a `Mutex`, `feed_summary()` for `list`, `now()` helper. `enclosures.url` is UNIQUE — the
|
||||
dedupe key that replaces `history.dat`.
|
||||
|
||||
`src/main.rs`: clap skeleton with `ipx list`. Only the subcommands that work exist; the rest arrive
|
||||
with their steps.
|
||||
|
||||
Verified: `cargo test` 5/5 green (config defaults, port-range fallback incl. reversed range,
|
||||
password_env precedence, schema idempotency, enclosure-url uniqueness). Gate met — `ipx --config
|
||||
<scratch> list` printed both feeds and created `state.db` once across two runs.
|
||||
|
||||
Deferred: nothing. Two dead-code warnings (`Config::save`, `Feed::password`) are expected; steps 3
|
||||
and 8 consume them.
|
||||
Next: step 3 — `feed.rs`.
|
||||
|
||||
Also added `chrono` 0.4 (std, clock) for RFC-2822 pubDate parsing in step 3.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-09 — Step 1: repo skeleton
|
||||
|
||||
Repo created at `/src/ipodderx-rs`, default branch `main`, `cargo init --name ipx` (edition 2024,
|
||||
rustc 1.95.0). LICENSE (MIT, carrying the 2010 copyright), README with the lineage note, and this
|
||||
file. Hello-world `main.rs` builds.
|
||||
|
||||
Dependency versions pinned today — the later steps are written against these APIs:
|
||||
|
||||
| crate | version | notes |
|
||||
|---|---|---|
|
||||
| tokio | 1.53.1 | rt-multi-thread, macros, fs, io-util, net, sync, time, signal |
|
||||
| reqwest | 0.13.5 | `default-features = false`; features **`rustls`** (not `rustls-tls` — renamed in 0.13), http2, gzip, stream, json, charset, **`system-proxy`** (env-var proxy pickup is opt-in in 0.13) |
|
||||
| rss | 2.1.1 | default features; the `with-syndication` feature name in my notes does not exist — Atom is handled by the separate crate |
|
||||
| atom_syndication | 0.12.10 | |
|
||||
| rusqlite | 0.40.2 | `bundled` |
|
||||
| librqbit | 9.0.1 | `default-features = false`; features `rust-tls`, `http-api-client`. The default `default-tls` feature pulls reqwest/native-tls **and** an OpenSSL sha1 backend (`crypto-hash`), which fails to build without pkg-config/OpenSSL headers. API not yet exercised — step 7 |
|
||||
| serde 1.0.229 / serde_json 1.0.151 / toml 1.1.5 | | |
|
||||
| clap | 4.6.6 | derive |
|
||||
| infer 0.22.0 / dirs 7.0.0 / anyhow 1.0.104 / opml 1.1.6 | | |
|
||||
| tracing 0.1.44 / tracing-subscriber 0.3.23 | | env-filter |
|
||||
|
||||
Deferred: nothing.
|
||||
|
||||
Gotcha worth keeping: three feature names in the plan were wrong against current crate versions —
|
||||
`reqwest/rustls-tls` is now `rustls`, env-var proxy support moved behind `system-proxy`, and
|
||||
`rss/with-syndication` does not exist. `librqbit`'s default features drag in OpenSSL; `rust-tls`
|
||||
is the fix. Whole tree is rustls-only now, no C TLS dependency.
|
||||
Next: step 2 — `config.rs` + `db.rs`. (done)
|
||||
150
README.md
150
README.md
@@ -1,111 +1,83 @@
|
||||
# ipodderx-rs
|
||||
|
||||
A headless podcatcher: scans RSS/Atom feeds, downloads enclosures (HTTP and BitTorrent),
|
||||
files them into per-feed folders, and reaps old episodes to stay under a disk quota.
|
||||
Runs as a one-shot CLI or as a daemon with a Unix-socket JSON event stream for a UI to attach to.
|
||||
A self-hosted podcatcher for a household. It checks your feeds, downloads the episodes, and serves
|
||||
a web UI modelled on the 2004 Mac app **iPodderX**, for any number of people sharing one copy of
|
||||
the files. One Rust binary, `ipx`, is both the daemon and the command line.
|
||||
|
||||
## Lineage
|
||||
It is a rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python engine
|
||||
behind iPodderX (2004-2008, Ray Slakinski & August Trometer).
|
||||
|
||||
This is a modern Rust rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the
|
||||
Python 2 engine behind **iPodderX** (2004-2008, Ray Slakinski & August Trometer), which was
|
||||
open-sourced under the MIT License in 2010.
|
||||
## What it does
|
||||
|
||||
What carries over: the feed scan and TTL handling, GUID/URL dedupe, per-feed and per-date download
|
||||
folders, keyword filters, the explicit-content filter, torrent enclosures, and "SmartSpace" -- the
|
||||
oldest-first disk quota reaper.
|
||||
- **The web UI.** It has a toolbar, and a feed list that opens with Directory, Popular and All
|
||||
Subscriptions. Items sit in a sortable table with a Files pane, and there is a player bar. It
|
||||
comes in Dark, Light and Classic themes, and works on a phone.
|
||||
- **Several people, one copy.** Each person has their own subscriptions and their own read, kept
|
||||
and playback state. There is one file on disk per episode, however many people want it. People
|
||||
sign in with a password or through a proxy (Cloudflare Zero Trust or Authentik), and admins
|
||||
manage accounts and settings.
|
||||
- **Scanning.** Feeds are checked on a schedule, globally or per feed, and a feed's own TTL is
|
||||
honoured. Keyword, explicit-content and media-type filters decide what is downloaded, with a cap
|
||||
on new downloads per scan.
|
||||
- **Downloads.** Files come over HTTP or BitTorrent and are filed into a folder per feed.
|
||||
Retention deletes the oldest files to stay under a disk quota or an age limit, and never touches
|
||||
an item someone has kept.
|
||||
- **OPML.** You can import and export your own subscriptions. You can also subscribe to an OPML
|
||||
URL, which keeps a whole list in step as a folder.
|
||||
|
||||
What does not: iTunes and iPhoto export via AppleScript, text-to-speech enclosures, the Windows
|
||||
WMP/COM paths, XML plists and Python pickles for state, the `directory.iPodderX.com` survey ping,
|
||||
3DES-encrypted preferences, and the `printMSG` stdout protocol (replaced by a JSON-lines socket).
|
||||
## Run it
|
||||
|
||||
## Quickstart
|
||||
With Docker:
|
||||
|
||||
```sh
|
||||
cargo install --path .
|
||||
|
||||
ipx add https://atp.fm/rss # names the feed from its own title
|
||||
ipx list
|
||||
ipx fetch # scan now
|
||||
ipx daemon # or run continuously, honouring each feed's <ttl>
|
||||
docker build -t ipodderx .
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Config lives at `~/.config/ipx/config.toml` (mode 0600, since it may hold feed passwords);
|
||||
state at `~/.local/share/ipx/state.db`. Override with `IPX_CONFIG` and `IPX_DATA_DIR`.
|
||||
Set `IPX_LOG=ipx=debug` for verbose logging on stderr.
|
||||
`docker-compose.yml` is set up for the author's own server. Point its `image` and its three volumes
|
||||
(`/config`, `/data` and `/downloads`) at yours first. The UI is on port 8099. BitTorrent uses 6881
|
||||
over TCP and UDP. Files are written as `PUID`/`PGID`, 99:100 by default.
|
||||
|
||||
## Commands
|
||||
From source:
|
||||
|
||||
| command | what it does |
|
||||
```sh
|
||||
cargo build --release
|
||||
./target/release/ipx daemon
|
||||
```
|
||||
|
||||
The first start creates **admin / ipodderx**. Sign in at `/login`, then change it:
|
||||
|
||||
```sh
|
||||
echo -n 'a good password' | ipx user passwd admin
|
||||
```
|
||||
|
||||
The UI is plain HTTP, so put TLS in front of it if it is reachable from outside your network.
|
||||
|
||||
## Documentation
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `ipx add <url> [--folder X] [--keywords a,b]` | subscribe; the id comes from the feed title |
|
||||
| `ipx rm <feed>` | unsubscribe; downloads and history are kept |
|
||||
| `ipx list` / `ipx status` | subscriptions and their state |
|
||||
| `ipx fetch [FEED] [--force]` | scan; `--force` ignores the TTL |
|
||||
| `ipx reap [--dry-run]` | run retention now |
|
||||
| `ipx import/export <file.opml>` | move subscriptions in or out |
|
||||
| `ipx daemon` | scheduler plus the control socket |
|
||||
| [docs/configuration.md](docs/configuration.md) | Every config key, path and environment variable |
|
||||
| [docs/cli.md](docs/cli.md) | Every command, including `ipx user` |
|
||||
| [docs/users.md](docs/users.md) | Accounts, and what several people share |
|
||||
| [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik |
|
||||
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API |
|
||||
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
|
||||
| [docs/history.md](docs/history.md) | How it was built, with what was wrong and why |
|
||||
| [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed |
|
||||
|
||||
Any command with a wire form probes the socket first: if a daemon is running it does the work,
|
||||
and the CLI just renders the events it streams back. `--local` forces in-process execution.
|
||||
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[general]
|
||||
download_dir = "~/Podcasts"
|
||||
socket = "/run/user/1000/ipx.sock" # default: $XDG_RUNTIME_DIR/ipx.sock
|
||||
interval_mins = 60 # default poll; a feed's own <ttl> wins when longer
|
||||
organize = "feed" # "feed" | "date"
|
||||
max_total_gb = 50 # 0 = unlimited
|
||||
max_age_days = 30 # 0 = keep forever
|
||||
|
||||
[torrent]
|
||||
enabled = true
|
||||
seed_ratio = 1.0 # stop seeding at this ratio ...
|
||||
seed_time_mins = 60 # ... or after this long, whichever comes first
|
||||
port_range = "6881-6889"
|
||||
stall_mins = 30 # give up on a torrent making no progress
|
||||
|
||||
[feeds.atp]
|
||||
url = "https://atp.fm/rss"
|
||||
folder = "Accidental Tech Podcast" # default: the feed title
|
||||
keywords = ["deep dive"] # OR across keywords, AND within one
|
||||
allow_explicit = false
|
||||
auto_download = true
|
||||
max_new_per_check = 3 # the rest wait for the next scan
|
||||
username = "ray" # optional HTTP basic auth
|
||||
password_env = "IPX_ATP_PASS" # or a literal `password`
|
||||
```
|
||||
|
||||
Retention keeps files that are `flagged` in the database, and deletes read episodes before unread
|
||||
ones, oldest first.
|
||||
|
||||
## Socket protocol
|
||||
|
||||
Newline-delimited JSON over a Unix socket, both directions.
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
$ printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
|
||||
{"ev":"feed_start","feed":"atp"}
|
||||
{"ev":"progress","feed":"atp","url":"...","file":"ep1.mp3","done":8192,"total":3000000}
|
||||
{"ev":"download_done","feed":"atp","url":"...","path":"...","bytes":3000000}
|
||||
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
|
||||
{"ev":"scan_done","feeds":1}
|
||||
cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state
|
||||
node tests/page-smoke.js # the page script loads without throwing
|
||||
npx playwright test # a real browser against a real daemon on fixture feeds
|
||||
```
|
||||
|
||||
Commands: `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `status`.
|
||||
Events: `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
|
||||
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
|
||||
`scan_done`, `reap_done` and `status` are terminal -- a client that asked for work stops there.
|
||||
|
||||
Progress is throttled to whole percents. The stream is a broadcast, so a client attached to a busy
|
||||
daemon also sees that daemon's other work.
|
||||
|
||||
## Running it as a service
|
||||
|
||||
`contrib/` has a systemd user unit for the daemon, and a timer plus one-shot service if you would
|
||||
rather run periodic scans with no daemon (in which case there is no socket for a UI to attach to).
|
||||
`npm install` gets the test runner, and `npx playwright install --with-deps chromium` gets the
|
||||
browser.
|
||||
|
||||
## License
|
||||
|
||||
MIT. See [LICENSE](LICENSE).
|
||||
MIT, see [LICENSE](LICENSE). The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by
|
||||
@fontawesome, under [CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG.
|
||||
|
||||
57
TODO.md
Normal file
57
TODO.md
Normal file
@@ -0,0 +1,57 @@
|
||||
# To do
|
||||
|
||||
## Errors in the log
|
||||
|
||||
From the production log and the feeds' stored errors on 2026-09-13. The Docker log only reached back
|
||||
to 12:32 UTC, so the list comes from `feeds.last_error`: 57 of 1,059 feeds. None of it is ipx's
|
||||
User-Agent; a browser gets the same answers.
|
||||
|
||||
- [x] **Stop scanning an OPML's feeds once nobody subscribes to it.** 55 of the 57 are feeds from
|
||||
`davewiner` (lists.opml.org/davefeeds.xml). The list left `config.toml` about 14 hours before this
|
||||
was written, but its 922 feeds are still in the database and still scanned every hour, with
|
||||
auto-download on: `subscriptions()` adds every derived feed, and with no parent to copy from,
|
||||
`auto_download` falls back to `true` (`src/main.rs:937`). `docs/users.md` says a feed stops being
|
||||
scanned when its last subscriber leaves. Skip a derived feed whose parent is not in config, and
|
||||
when the last subscriber leaves an OPML, treat its feeds the way `sync_group` treats ones the list
|
||||
dropped: remove those with nothing downloaded, mark the rest orphaned. `remove_feed` and `ipx rm`
|
||||
both leave them behind today. (`src/main.rs`, `src/web.rs`)
|
||||
- [x] **Read feeds with a bare `&`.** kcpw has `<link>https://kcpw.org/?post_type=post&p=125715</link>`,
|
||||
and both feedland feeds have the same fault. Strict XML refuses them; browsers and other readers do
|
||||
not. When `feed::parse` fails, try once more with every `&` that does not start an entity written
|
||||
as `&`. Nobody subscribes to these three now, but the next feed like them will fail the same
|
||||
way. (`src/feed.rs`)
|
||||
- [x] **Say what came back when it is not a feed.** Thirteen errors read "not RSS (the input did not
|
||||
begin with an rss tag) and not Atom (...)". Each one checked was a web page: the feed moved and its
|
||||
old URL redirects to the site, or the domain lapsed. Say "got a web page, not a feed", and when the
|
||||
page links a feed (`<link rel="alternate" type="application/rss+xml">`), name it. That link found
|
||||
the new feed for om.co, ms.now, Letters of Note, the Daily Dot, Hell Gate, The Frame Lab and
|
||||
Daily Kos. A `202` with an empty body (British Antarctic Survey) should read as "nothing yet", not
|
||||
as a parse failure. (`src/feed.rs`)
|
||||
- [x] **Show a publisher's error in the UI.** Today a failing feed shows its raw error in red only
|
||||
once you open it (`web/index.html:877`, `:960`); the OPML view marks a failing child "error"
|
||||
(`:988`), and the sidebar shows nothing. Mark a failing feed in the sidebar too, and say whose
|
||||
problem it is and what to do, in plain words: a 404 means the publisher took the feed down or moved
|
||||
it, 401 and 403 that the site refuses ipx, 402 that the feed needs a paid plan, a name that no
|
||||
longer resolves that the site is gone, and a web page instead of a feed that it moved (with the new
|
||||
URL when the page links one). Offer Unsubscribe, or Use the new address. Keep timeouts, 5xx and
|
||||
429 out of it, and flag a feed only once it has failed for a day: macmanx failed once and read
|
||||
fine an hour later. That needs the time a feed started failing, kept beside `last_error` and
|
||||
cleared when a check succeeds. (`web/index.html`, `src/db.rs`, `src/main.rs`)
|
||||
- [ ] **Westenberg.** The only failing feed anyone subscribes to. The Hacker News stars list still
|
||||
gives `joanwestenberg.com/rss`, which is a 404; the feed is now `joanwestenberg.com/feed`.
|
||||
Nothing for ipx to fix; subscribe to the new URL directly.
|
||||
|
||||
## Other Fixes and Features
|
||||
|
||||
- [ ] Remember which feed is selected and view (all, unread, flagged, etc) user as selected between visits. If unknown default to All Subscriptions
|
||||
- [x] When clicking any link it should open in a new tab
|
||||
- [ ] In mobile (iOS) sometimes the top line items like the hamburger menu are not clickable unless you do a hard refresh
|
||||
- [x] video files play as audio files, they should play as video.
|
||||
- [ ] Move Light/Dark/Classic options to user settings. Include an Auto mode that uses system preferences for light/dark modes
|
||||
- [ ] Below Popular, have a currently listening section to show what podcasts have been started and not finnished
|
||||
- [x] Update subscribe/unsubscribe icons to be circle-minus (unsubscribe) and circle-check (subscribe)
|
||||
- [x] If I'm on the Unread tab, and I click to read an item the entry in the list will disappear. it should remain until I click to another item.
|
||||
|
||||
## Directory Overhaul
|
||||
|
||||
- [ ] Directory needs to be more functional, with categories and a more interesting layout. use /frontend-design to help
|
||||
@@ -1,7 +0,0 @@
|
||||
[Unit]
|
||||
Description=ipx feed scan (one shot)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=%h/.cargo/bin/ipx fetch
|
||||
Environment=IPX_LOG=ipx=info
|
||||
@@ -1,18 +0,0 @@
|
||||
# User unit: install to ~/.config/systemd/user/ipx.service, then
|
||||
# systemctl --user enable --now ipx
|
||||
# The socket lands in $XDG_RUNTIME_DIR/ipx.sock by default, so a UI running as the
|
||||
# same user can attach without extra configuration.
|
||||
[Unit]
|
||||
Description=ipx podcatcher
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=%h/.cargo/bin/ipx daemon
|
||||
Restart=on-failure
|
||||
RestartSec=30
|
||||
Environment=IPX_LOG=ipx=info
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -1,16 +0,0 @@
|
||||
# Alternative to the daemon: a periodic one-shot scan, closer to how the original
|
||||
# iPodderX agent was driven. Use this OR ipx.service, not both -- with no daemon
|
||||
# running there is no socket, so a UI cannot attach.
|
||||
#
|
||||
# Install ipx-scan.service and ipx.timer to ~/.config/systemd/user/, then
|
||||
# systemctl --user enable --now ipx.timer
|
||||
[Unit]
|
||||
Description=Periodic ipx feed scan
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
OnUnitActiveSec=1h
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
24
docker-compose.yml
Normal file
24
docker-compose.yml
Normal file
@@ -0,0 +1,24 @@
|
||||
services:
|
||||
ipodderx:
|
||||
image: 192.168.1.130:5000/ipodderx:latest
|
||||
container_name: iPodderX
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
PUID: "99"
|
||||
PGID: "100"
|
||||
TZ: "America/Toronto"
|
||||
IPX_LOG: "ipx=info"
|
||||
ports:
|
||||
- "8099:8099" # web UI
|
||||
- "6881:6881/tcp" # BitTorrent peers
|
||||
- "6881:6881/udp" # DHT
|
||||
volumes:
|
||||
- /mnt/fast/appdata/ipodderx:/config # config.toml, and the web token
|
||||
- /mnt/user/ipodderx/:/data # state.db
|
||||
- /mnt/user/ipodderx/downloads:/downloads
|
||||
healthcheck:
|
||||
test: ["CMD", "ipx", "status"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
40
docker-entrypoint.sh
Executable file
40
docker-entrypoint.sh
Executable file
@@ -0,0 +1,40 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
# A container's loopback is not reachable from outside it, so the default bind of
|
||||
# 127.0.0.1 would leave the UI unreachable. Write a starter config that binds 0.0.0.0
|
||||
# on first run; after that the file is yours and is never rewritten.
|
||||
if [ ! -f "$IPX_CONFIG" ]; then
|
||||
mkdir -p "$(dirname "$IPX_CONFIG")"
|
||||
cat > "$IPX_CONFIG" <<TOML
|
||||
[general]
|
||||
download_dir = "/downloads"
|
||||
schedule = "every 60m"
|
||||
max_total_gb = 0
|
||||
max_age_days = 0
|
||||
|
||||
[torrent]
|
||||
enabled = true
|
||||
port_range = "6881-6889"
|
||||
|
||||
[web]
|
||||
enabled = true
|
||||
bind = "0.0.0.0:8099"
|
||||
token = ""
|
||||
TOML
|
||||
echo "ipx: wrote a starter config to $IPX_CONFIG"
|
||||
fi
|
||||
|
||||
mkdir -p "$IPX_DATA_DIR" /downloads
|
||||
|
||||
# Unraid shares expect 99:100. Running as root would leave root-owned downloads.
|
||||
if [ "$(id -u)" = "0" ] && [ -n "$PUID" ] && [ -n "$PGID" ]; then
|
||||
if ! getent group ipx >/dev/null 2>&1; then addgroup --gid "$PGID" ipx 2>/dev/null || true; fi
|
||||
if ! getent passwd ipx >/dev/null 2>&1; then
|
||||
adduser --uid "$PUID" --gid "$PGID" --disabled-password --gecos "" ipx 2>/dev/null || true
|
||||
fi
|
||||
chown -R "$PUID:$PGID" "$IPX_DATA_DIR" "$(dirname "$IPX_CONFIG")" 2>/dev/null || true
|
||||
exec gosu "$PUID:$PGID" "$@"
|
||||
fi
|
||||
|
||||
exec "$@"
|
||||
146
docs/architecture.md
Normal file
146
docs/architecture.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# How it works
|
||||
|
||||
One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a Unix-socket
|
||||
control server, and the web UI. Everything else is a CLI that either does the work itself or hands
|
||||
it to a running daemon.
|
||||
|
||||
## Modules
|
||||
|
||||
| File | Responsibility | What it replaced in the Python |
|
||||
|---|---|---|
|
||||
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
|
||||
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
|
||||
| `src/db.rs` | SQLite schema, migrations, every query | `.ipxd` plists, `history.dat`, `qmcache.dat` |
|
||||
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
|
||||
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
|
||||
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
|
||||
| `src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` |
|
||||
| `src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout |
|
||||
| `src/auth.rs` | Argon2id hashing, session tokens, header names | — |
|
||||
| `src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — |
|
||||
| `src/logbuf.rs` | Ring buffer behind the UI's Log view | — |
|
||||
| `web/index.html` | The whole front end, `include_str!`d into the binary | — |
|
||||
|
||||
The page is compiled in, so **editing `web/index.html` needs a rebuild**.
|
||||
|
||||
## A scan
|
||||
|
||||
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
|
||||
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
|
||||
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
|
||||
re-derived into the database each scan, never written to config.toml. A Patreon creator link
|
||||
(a token, no `show=`) with more than one show is treated the same way, before any fetch: its
|
||||
shows come from Patreon's web API and each becomes a derived feed.
|
||||
4. Record entries. A changed title or description flips the item back to unread.
|
||||
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
|
||||
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
|
||||
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
|
||||
a reason. What a filter skipped is judged again every scan, so a change of settings brings it
|
||||
back. A feed in a group takes your settings on the group for anything you have not set on it.
|
||||
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
|
||||
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
|
||||
or an error page — and is deleted.
|
||||
|
||||
## Data model
|
||||
|
||||
```
|
||||
feeds id, url, title, image, etag, last_modified, last_checked, ttl_mins,
|
||||
last_error, orphaned, group_id, managed
|
||||
entries feed_id, guid, title, link, published, description, first_seen,
|
||||
image, duration, episode, season PK (feed_id, guid)
|
||||
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,
|
||||
bytes_done, downloaded_at, last_error
|
||||
users id, name, pass_hash, is_admin, created, last_login
|
||||
sessions token, user_id, seen
|
||||
subscriptions user_id, feed_id, keywords, auto_download, allow_explicit,
|
||||
max_new_per_check PK (user_id, feed_id)
|
||||
entry_state user_id, feed_id, guid, read, flagged, position
|
||||
PK (user_id, feed_id, guid)
|
||||
```
|
||||
|
||||
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
|
||||
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
|
||||
older database.
|
||||
|
||||
Schema changes: add the table or column to `SCHEMA`. `CREATE TABLE IF NOT EXISTS` leaves a table
|
||||
that already exists alone, so a new column on one also goes in `migrate()`'s `wanted` list, and a
|
||||
retired one in its `retired` list; both are checked with `PRAGMA table_info`. Columns from before
|
||||
0.3.0, the oldest version an upgrade may start from, need no entry. `Db::memory()` runs the same
|
||||
path as `Db::open`, so a migration cannot pass the tests while missing in production.
|
||||
|
||||
## Control socket
|
||||
|
||||
Newline-delimited JSON, both directions, over `$XDG_RUNTIME_DIR/ipx.sock`.
|
||||
|
||||
```sh
|
||||
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
|
||||
{"ev":"feed_start","feed":"atp"}
|
||||
{"ev":"progress","feed":"atp","enclosure":42,"file":"ep1.mp3","done":8192,"total":3000000}
|
||||
{"ev":"download_done","feed":"atp","enclosure":42,"path":"…","bytes":3000000}
|
||||
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
|
||||
{"ev":"scan_done","feeds":1}
|
||||
```
|
||||
|
||||
**Commands** — `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `download`
|
||||
(`enclosure`), `status`.
|
||||
|
||||
**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
|
||||
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
|
||||
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
|
||||
there. Commands run one at a time, in the order they arrive, except `status`: the socket answers it
|
||||
straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and
|
||||
answers only the client that asked, since `status` would end any other client's session.
|
||||
|
||||
Progress carries the enclosure id, without which a UI cannot tell one download from another and
|
||||
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
|
||||
so a client attached to a busy daemon also sees that daemon's other work.
|
||||
|
||||
Inside the process the same events go over a `tokio::broadcast`; commands arrive on an `mpsc` and
|
||||
are handled by a single worker, so nothing races over the same download. Shutdown is a `watch`
|
||||
channel raced *inside* each job — `tokio::select!` only races branches at the point of selection,
|
||||
so a long download had to be able to notice the signal itself.
|
||||
|
||||
## HTTP API
|
||||
|
||||
Everything below `/api` needs a signed-in user; the browser gets a redirect to `/login`, anything
|
||||
else a `401`.
|
||||
|
||||
| Route | |
|
||||
|---|---|
|
||||
| `GET /` | the app |
|
||||
| `GET /login`, `POST /api/login`, `POST /api/logout`, `GET /api/me` | sign-in |
|
||||
| `GET /api/feeds`, `POST /api/feeds` | your subscriptions; subscribe |
|
||||
| `PATCH /api/feeds/{id}`, `DELETE /api/feeds/{id}` | your settings or (admin) the feed's; unsubscribe |
|
||||
| `GET /api/feeds/{id}/entries` | paged, filtered, searchable, sortable (`sort` = kept, title, feed, type, size or published; `dir` = asc or desc) |
|
||||
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
|
||||
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
|
||||
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
|
||||
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, kept, position |
|
||||
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
|
||||
| `POST /api/fetch` | |
|
||||
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
|
||||
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with an OPML's feeds in place of the OPML and everyone counted (id, title, art, count, whether it is yours; never a URL, never a private feed); subscribe by id |
|
||||
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
|
||||
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |
|
||||
| `GET /api/events` | SSE, the same broadcast the socket carries |
|
||||
| `GET /api/logs` | admin-only; the ring buffer, with a sequence cursor |
|
||||
| `GET /media/{id}` | the file, with Range support so seeking works |
|
||||
|
||||
Show notes are feed-supplied HTML from an untrusted source, sanitized with `ammonia` server-side
|
||||
before they reach the page.
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
cargo test # parsing, filters, retention, schedules, SQL, per-user isolation
|
||||
node tests/page-smoke.js # the page script loads and every selector it wires at load exists
|
||||
npx playwright test # a real browser against a real daemon on fixture feeds
|
||||
```
|
||||
|
||||
The Rust tests cannot see a wrong selector, a handler that runs and does nothing, or a page that
|
||||
renders empty — which is what has actually reached users. Each Playwright case maps to a bug that
|
||||
did.
|
||||
|
||||
The suite starts its own daemon and database under `/tmp/ipx-ui-test`, wiped once per run. Tests
|
||||
share that daemon and run in order, so a test that marks something read changes what later tests
|
||||
see — make assertions that do not depend on earlier ones.
|
||||
91
docs/cli.md
Normal file
91
docs/cli.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Command line
|
||||
|
||||
```
|
||||
ipx [--config PATH] [--local] <command>
|
||||
```
|
||||
|
||||
Every command that has a wire form probes the control socket first: if a daemon is running, the
|
||||
daemon does the work and the CLI just renders the events it streams back. That is deliberate — two
|
||||
processes must never download the same thing. `--local` forces the work to happen in-process.
|
||||
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
| `ipx list` | Subscriptions and their state |
|
||||
| `ipx status` | Counts: feeds, pending, downloaded |
|
||||
| `ipx fetch [FEED] [--force]` | Scan everything, or one feed. `--force` ignores the TTL |
|
||||
| `ipx add <url> [--folder X] [--keywords a,b]` | Subscribe; the id comes from the feed title |
|
||||
| `ipx rm <feed>` | Unsubscribe; downloads and history are kept |
|
||||
| `ipx import <file.opml>` / `ipx export <file.opml>` | Move subscriptions in or out. Import subscribes the first admin, as the shared web token does; in the web UI it subscribes whoever is signed in |
|
||||
| `ipx reap [--dry-run]` | Run retention now |
|
||||
| `ipx user <add\|list\|passwd\|rm>` | Accounts for the web UI |
|
||||
| `ipx daemon [--web ADDR]` | Scheduler, control socket and web UI |
|
||||
|
||||
## Accounts
|
||||
|
||||
Passwords are read from **stdin**, so they miss the shell history and any `ps` listing.
|
||||
|
||||
```sh
|
||||
echo -n 'a good password' | ipx user add ray # local account
|
||||
ipx user add ray@example.com --no-password # signs in through the proxy only
|
||||
echo -n 'a good password' | ipx user passwd admin # change a password
|
||||
ipx user list # who exists, and how each signs in
|
||||
ipx user rm sam # account, subscriptions and read state
|
||||
```
|
||||
|
||||
The first account created is an admin; later ones are ordinary users. A database with no accounts
|
||||
at all gets **admin / ipodderx** on the next daemon start, announced in the log — change it.
|
||||
|
||||
To avoid even the command line, read it interactively:
|
||||
|
||||
```sh
|
||||
read -s PW && echo -n "$PW" | ipx user passwd admin
|
||||
```
|
||||
|
||||
## Scanning
|
||||
|
||||
```sh
|
||||
ipx fetch # everything due
|
||||
ipx fetch atp --force # one feed, ignoring its TTL and schedule
|
||||
```
|
||||
|
||||
A scan: conditional GET (`If-None-Match` / `If-Modified-Since`), parse, record new entries, apply
|
||||
the filters, then download up to the per-scan cap, newest first. A feed nothing has changed in
|
||||
answers `304` and costs one request.
|
||||
|
||||
## Retention
|
||||
|
||||
```sh
|
||||
ipx reap --dry-run # what would go, oldest first
|
||||
ipx reap # actually delete
|
||||
```
|
||||
|
||||
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
|
||||
with no file are pruned from the database. **An item anyone kept keeps its file**, and one only counts
|
||||
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
|
||||
stops the next scan fetching it again.
|
||||
|
||||
## The daemon
|
||||
|
||||
```sh
|
||||
ipx daemon # scheduler + socket + web UI
|
||||
ipx daemon --web 0.0.0.0:8099 # override the configured bind for one run
|
||||
```
|
||||
|
||||
One daemon per socket; a second refuses to start rather than fight over the database. It shuts down
|
||||
cleanly on SIGTERM, including mid-download.
|
||||
|
||||
To kill it, match the binary exactly:
|
||||
|
||||
```sh
|
||||
pkill -x ipx
|
||||
```
|
||||
|
||||
`pkill -f ipx` matches the shell running the command too, and kills your own session.
|
||||
|
||||
## Talking to it directly
|
||||
|
||||
```sh
|
||||
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
|
||||
```
|
||||
|
||||
See [architecture.md](architecture.md#control-socket) for the protocol.
|
||||
120
docs/configuration.md
Normal file
120
docs/configuration.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Configuration
|
||||
|
||||
One TOML file, read at startup and re-read whenever the web UI writes to it — most changes take
|
||||
effect without a restart. Default location `$XDG_CONFIG_HOME/ipx/config.toml`
|
||||
(`~/.config/ipx/config.toml`), overridden with `--config` or `$IPX_CONFIG`.
|
||||
|
||||
| What | Where | Override |
|
||||
|---|---|---|
|
||||
| Config | `~/.config/ipx/config.toml` | `--config`, `$IPX_CONFIG` |
|
||||
| Database | `~/.local/share/ipx/state.db` | `$IPX_DATA_DIR` |
|
||||
| Control socket | `$XDG_RUNTIME_DIR/ipx.sock` | `[general] socket` |
|
||||
| Downloads | `[general] download_dir` | — |
|
||||
|
||||
`~` is expanded in paths. The database is SQLite in WAL mode; back it up by copying `state.db`
|
||||
while the daemon is stopped, or with `sqlite3 state.db .backup`.
|
||||
|
||||
## `[general]`
|
||||
|
||||
```toml
|
||||
[general]
|
||||
download_dir = "~/Podcasts"
|
||||
socket = "/run/user/1000/ipx.sock"
|
||||
schedule = "every 1h" # "every 30m", "every 4h", "2d", "90" (minutes)
|
||||
organize = "feed" # "feed" | "date"
|
||||
max_total_gb = 50 # 0 = unlimited
|
||||
max_age_days = 30 # 0 = keep forever
|
||||
max_new_per_check = 3 # per feed, per scan. 0 = unlimited
|
||||
media_types = ["audio", "video"]
|
||||
```
|
||||
|
||||
* **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to
|
||||
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
|
||||
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
|
||||
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
|
||||
pad. Kept items are never deleted, and a file only counts as read once every subscriber has
|
||||
read it. `0` disables it entirely.
|
||||
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
|
||||
Kept ones stay. `0` disables it.
|
||||
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
|
||||
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
|
||||
fetched 216 files and 22 GB in one scan.
|
||||
* **`media_types`** — top-level MIME types taken automatically. Anything else is still listed and
|
||||
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
|
||||
without this the disk fills with artwork. Empty takes everything.
|
||||
|
||||
## `[torrent]`
|
||||
|
||||
```toml
|
||||
[torrent]
|
||||
enabled = true
|
||||
seed_ratio = 1.0 # stop seeding at this ratio ...
|
||||
seed_time_mins = 60 # ... or after this long, whichever comes first
|
||||
port_range = "6881-6889"
|
||||
stall_mins = 30 # give up on a torrent making no progress
|
||||
```
|
||||
|
||||
A `.torrent` body is handed to the torrent path whatever MIME type it was advertised as. Torrents
|
||||
run on their own tasks (two at a time) so a slow swarm never blocks a scan.
|
||||
|
||||
## `[web]`
|
||||
|
||||
```toml
|
||||
[web]
|
||||
enabled = true
|
||||
bind = "0.0.0.0:8099" # 127.0.0.1:8080 by default
|
||||
token = "" # generated and saved on first run
|
||||
trusted_header = "" # e.g. "Cf-Access-Authenticated-User-Email"
|
||||
trusted_proxies = ["127.0.0.1", "::1"]
|
||||
auto_create_users = true
|
||||
sign_out_url = "" # e.g. "/cdn-cgi/access/logout"
|
||||
session_days = 30
|
||||
```
|
||||
|
||||
* **`token`** — the shared secret, which signs in as the **admin**. `?token=…` sets a cookie, so
|
||||
you paste it once per browser. It is what the Docker healthcheck and any scripts use.
|
||||
* **`trusted_header`** — a header naming the signed-in user, set by whatever fronts ipx. Empty
|
||||
disables that path. See [sso.md](sso.md).
|
||||
* **`trusted_proxies`** — addresses allowed to assert that header, and the entire security boundary
|
||||
for it. Name the proxy, never a subnet.
|
||||
* **`auto_create_users`** — create an account the first time the proxy vouches for a new name.
|
||||
* **`sign_out_url`** — where Sign out sends someone the proxy signed in: the proxy's own sign-out,
|
||||
`/cdn-cgi/access/logout` behind Cloudflare Access. Empty sends them to the sign-in page, where
|
||||
the proxy signs them straight back in.
|
||||
* **`session_days`** — sign a session out after this long without a request.
|
||||
|
||||
It is plain HTTP. On a LAN bind everything crosses the network in the clear — and a feed URL can
|
||||
itself carry a credential. Put TLS in front of it if that matters.
|
||||
|
||||
## `[feeds.<id>]`
|
||||
|
||||
The table key is the feed id: stable, human-readable, and used in paths and the API. `ipx add`
|
||||
derives it from the feed title.
|
||||
|
||||
```toml
|
||||
[feeds.atp]
|
||||
url = "https://atp.fm/rss"
|
||||
folder = "Accidental Tech Podcast" # default: the feed title
|
||||
schedule = "every 6h" # overrides [general] for this feed
|
||||
media_types = ["audio"] # overrides [general] for this feed
|
||||
username = "ray" # HTTP basic auth
|
||||
password_env = "IPX_ATP_PASS" # preferred over a literal `password`
|
||||
```
|
||||
|
||||
With more than one account, **`keywords`, `auto_download`, `allow_explicit` and
|
||||
`max_new_per_check` live on each person's subscription in the database**, not here — the values in
|
||||
config.toml are the fallback for a feed nobody has claimed. The keys above describe the feed itself
|
||||
and are the same for everyone. See [users.md](users.md).
|
||||
|
||||
Feeds derived from a subscribed OPML are **not** written here: the OPML is the source of truth and
|
||||
they are re-derived on every scan. Editing one in the UI promotes it to a real config entry.
|
||||
|
||||
## Environment
|
||||
|
||||
| Variable | Effect |
|
||||
|---|---|
|
||||
| `IPX_CONFIG` | Config file path |
|
||||
| `IPX_DATA_DIR` | Directory holding `state.db` |
|
||||
| `IPX_LOG` | What reaches stderr (`ipx=debug`, `ipx::scan=debug`, …) |
|
||||
| `IPX_UI_LOG` | What the in-process log buffer captures for the UI's Log view |
|
||||
| `http_proxy` / `https_proxy` | Honoured for feed and enclosure fetches |
|
||||
1670
docs/history.md
Normal file
1670
docs/history.md
Normal file
File diff suppressed because it is too large
Load Diff
203
docs/sso.md
Normal file
203
docs/sso.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# Signing in through Cloudflare Access and Authentik
|
||||
|
||||
ipx can take the signed-in identity from whatever sits in front of it, instead of asking for a
|
||||
password itself. The proxy authenticates the person and passes the result to ipx in a **header**;
|
||||
ipx reads it, finds (or creates) the matching account, and gets on with it.
|
||||
|
||||
Read [How this is secured](#how-this-is-secured) before exposing anything. The short version: a
|
||||
header is worth exactly as much as the hop that set it, so ipx only believes one from an address you
|
||||
list.
|
||||
|
||||
---
|
||||
|
||||
## How ipodderx.sdf1.net does it
|
||||
|
||||
Checked end to end on 2026-09-12. An earlier version of this page had never been tried against a
|
||||
real setup and pointed at the wrong address.
|
||||
|
||||
```
|
||||
browser ─► Cloudflare Access, app "ipodderx" ─── sign in ───► Authentik (OpenID Connect)
|
||||
─► tunnel "rays-unraid" (the cloudflared container on Tower)
|
||||
─► http://192.168.1.130:8099 ─► ipx
|
||||
```
|
||||
|
||||
Authentik is not in the request path. It is the identity provider Cloudflare Access asks. Access
|
||||
then adds `Cf-Access-Authenticated-User-Email`, the email address Authentik gave it, to every
|
||||
request it forwards through the tunnel, and ipx signs that person in.
|
||||
|
||||
| Piece | Where | Setting |
|
||||
|---|---|---|
|
||||
| Identity provider | Zero Trust → Settings → Authentication | `Authentik`, OpenID Connect; scopes `openid email profile` |
|
||||
| Access application | Zero Trust → Access → Applications → `ipodderx` | Domain `ipodderx.sdf1.net`; identity providers: Authentik only, with instant auth; session 730h; policy *Require Login* allows a list of email addresses |
|
||||
| Tunnel route | Zero Trust → Networks → Tunnels → `rays-unraid` → Public hostnames | `ipodderx.sdf1.net` → HTTP `192.168.1.130:8099` |
|
||||
| DNS | `sdf1.net` | `ipodderx` CNAME to the tunnel, proxied |
|
||||
| ipx | `/mnt/fast/appdata/ipodderx/config.toml`, `[web]` | below |
|
||||
|
||||
```toml
|
||||
[web]
|
||||
enabled = true
|
||||
bind = "0.0.0.0:8099"
|
||||
trusted_header = "Cf-Access-Authenticated-User-Email"
|
||||
trusted_proxies = ["127.0.0.1", "::1", "192.168.16.1"]
|
||||
auto_create_users = true
|
||||
sign_out_url = "/cdn-cgi/access/logout"
|
||||
session_days = 30
|
||||
```
|
||||
|
||||
Restart ipx after editing it: `docker compose -f /mnt/fast/arcane/projects/content/compose.yaml
|
||||
restart ipodderx`.
|
||||
|
||||
### What was missing
|
||||
|
||||
Cloudflare and Authentik were already right. Three things on the ipx side were not:
|
||||
|
||||
1. **`trusted_header` was empty**, which switches the whole proxy path off. ipx ignored the header
|
||||
and asked for a password.
|
||||
2. **`trusted_proxies` listed only `127.0.0.1`.** The tunnel's requests do not come from there;
|
||||
see the next section.
|
||||
3. **The account had the wrong name.** It was made by hand as `rays`, but the header carries
|
||||
`rays@sdf1.net`. With `auto_create_users` on, the first visit would have made a second, empty
|
||||
account. `ipx user rename rays rays@sdf1.net` fixed that without losing anything.
|
||||
|
||||
### The address to trust, and why it is 192.168.16.1
|
||||
|
||||
`cloudflared` runs in its own container and reaches ipx through the host's published port. Docker
|
||||
(iptables firewall backend) masquerades traffic between its bridge networks, so the tunnel's
|
||||
requests arrive from the **gateway of ipx's own network**, `content_default`:
|
||||
|
||||
```sh
|
||||
docker network inspect content_default -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'
|
||||
```
|
||||
|
||||
That was measured, not assumed. ipx does not log where a request came from, so the addresses were
|
||||
read from the kernel's connection table inside the container while the site was open. (`/proc/net/tcp`
|
||||
lists them in hex.)
|
||||
|
||||
If the `content` project's network is ever recreated, its gateway can change. Check it again, and
|
||||
update `trusted_proxies` to match.
|
||||
|
||||
### Names
|
||||
|
||||
The username is the email address, lower-cased: `rays@sdf1.net`. To sign in at `/login` with a
|
||||
password from the LAN, use that name too.
|
||||
|
||||
To let someone else in, add their address to the Access policy; they need an Authentik account with
|
||||
that email. With `auto_create_users = true` they get an ipx account on their first visit, as an
|
||||
ordinary user with no feeds. An account made before the proxy can be given the name the proxy will
|
||||
send:
|
||||
|
||||
```sh
|
||||
docker exec iPodderX ipx user rename <old name> <email address>
|
||||
```
|
||||
|
||||
### Signing out
|
||||
|
||||
**Sign out** sends someone the proxy signed in to `sign_out_url`, here Cloudflare's
|
||||
`/cdn-cgi/access/logout`. That ends your Access session for **every** Access application,
|
||||
`code.sdf1.net` included: Cloudflare has no way to end just one, and its sign-out page does not send
|
||||
you anywhere afterwards. The next visit goes back through Authentik, which lets you straight in if
|
||||
you are still signed in there. Signing out of Authentik itself is Authentik's own sign-out.
|
||||
|
||||
ipx never shows its password page to someone the proxy vouches for: `/login` sends them on to their
|
||||
feeds.
|
||||
|
||||
### The tile in Authentik's library
|
||||
|
||||
Authentik's library lists Authentik's own applications, and ipodderx signs in through the one
|
||||
called `Cloudflare Access`, so ipodderx needs a bookmark of its own to show up there. It is
|
||||
Applications → Applications → `ipodderx`: no provider, launch URL `https://ipodderx.sdf1.net`, and
|
||||
the iPodderX icon. Like Outline's, it has no policy bindings, so everyone in Authentik sees the
|
||||
tile. Who actually gets in is still up to the Access policy.
|
||||
|
||||
### Check it
|
||||
|
||||
```sh
|
||||
# From Tower itself: not a trusted address, so the header is ignored.
|
||||
curl -s -H 'Accept: application/json' -H 'Cf-Access-Authenticated-User-Email: rays@sdf1.net' \
|
||||
http://192.168.1.130:8099/api/me # -> sign in
|
||||
|
||||
# From a container on a Docker bridge, as cloudflared is: believed.
|
||||
docker run --rm --network bridge mirror.gcr.io/library/busybox wget -qO- \
|
||||
--header 'Accept: application/json' --header 'Cf-Access-Authenticated-User-Email: rays@sdf1.net' \
|
||||
http://192.168.1.130:8099/api/me # -> {"admin":true,"name":"rays@sdf1.net"}
|
||||
```
|
||||
|
||||
Then open `https://ipodderx.sdf1.net` in a private window. Authentik should ask who you are, and
|
||||
ipx should show `rays@sdf1.net` in the sidebar footer without asking for a password.
|
||||
|
||||
---
|
||||
|
||||
## The ipx settings
|
||||
|
||||
| Key | What it does |
|
||||
|---|---|
|
||||
| `trusted_header` | The header the proxy sets. Empty, the default, turns the proxy path off. |
|
||||
| `trusted_proxies` | The addresses allowed to set it. Nothing else is believed. |
|
||||
| `auto_create_users` | Make an account the first time the proxy vouches for a name ipx has not seen. |
|
||||
| `sign_out_url` | Where Sign out sends someone the proxy signed in: the proxy's own sign-out. Empty sends them to the sign-in page, where the proxy signs them straight back in. |
|
||||
| `session_days` | How long a password sign-in lasts without use. |
|
||||
|
||||
The first account ever created is an admin. Every later one is an ordinary user, who cannot change
|
||||
global settings, a feed's URL or folder, or how often feeds are scanned: the API refuses those with
|
||||
a `403`, not just the UI. Everything else about a feed is theirs alone; see [users.md](users.md).
|
||||
|
||||
Local sign-in at `/login` keeps working alongside the proxy, which is how you get in from the LAN
|
||||
when the tunnel is down. So does the shared `[web] token`, which signs in as the admin and is the
|
||||
way back in if you lock yourself out. A brand new database starts with **admin / ipodderx**;
|
||||
change it.
|
||||
|
||||
---
|
||||
|
||||
## Authentik in the request path instead
|
||||
|
||||
Not what ipodderx.sdf1.net uses, and **not verified**. Authentik can also sit in front of ipx
|
||||
itself, with a **Proxy Provider** and an **outpost** that adds `X-authentik-username`:
|
||||
|
||||
- Applications → Providers → Create → Proxy Provider; mode **Proxy** (the outpost talks to ipx) or
|
||||
**Forward auth** (an existing reverse proxy asks the outpost).
|
||||
- Applications → Create, bound to that provider, with a policy; add the provider to an outpost.
|
||||
- In ipx: `trusted_header = "X-authentik-username"`, and the outpost's or reverse proxy's address
|
||||
in `trusted_proxies`. Measure that address as above rather than guessing it.
|
||||
|
||||
---
|
||||
|
||||
## How this is secured
|
||||
|
||||
**The header is only believed from `trusted_proxies`.** Every other source is ignored, and the
|
||||
request falls through to a session cookie or the shared token. That is the whole security boundary.
|
||||
|
||||
With the tunnel reaching ipx through the host's port, `192.168.16.1` means **any container on Tower
|
||||
that connects to `192.168.1.130:8099`**, not only `cloudflared`. Machines on the LAN, and Tower
|
||||
itself, arrive under their own addresses and cannot set the header; the checks above show both
|
||||
sides. Never list a LAN address or range: anyone there could then send
|
||||
`Cf-Access-Authenticated-User-Email: rays@sdf1.net` and be you.
|
||||
|
||||
**What ipx does not do:** it does not verify Cloudflare's signed `Cf-Access-Jwt-Assertion`. It
|
||||
trusts the hop. Verifying the signature would make the containers on Tower irrelevant to the
|
||||
boundary, and is the upgrade if that ever matters.
|
||||
|
||||
**Turning it off:** clear `trusted_header` and restart. Proxy-made accounts stay, but nobody can sign
|
||||
in with them until they are given a password (`ipx user passwd <name>`).
|
||||
|
||||
---
|
||||
|
||||
## Everyday administration
|
||||
|
||||
```sh
|
||||
ipx user list # who exists, how each signs in, and when
|
||||
echo -n 'secret123' | ipx user add sam # local account, password on stdin
|
||||
ipx user add sam@example.com --no-password # proxy-only account, made ahead of time
|
||||
ipx user rename sam sam@example.com # give an account the name the proxy sends
|
||||
echo -n 'newsecret' | ipx user passwd sam # change a password
|
||||
ipx user rm sam # remove the account
|
||||
```
|
||||
|
||||
In the container, put `docker exec iPodderX` in front, and `docker exec -i iPodderX` for the ones
|
||||
that read a password.
|
||||
|
||||
Set `auto_create_users = false` once everyone who should have an account has one. After that the
|
||||
proxy vouching for an unknown name is logged and refused. Make people ahead of time instead, with
|
||||
the exact name the header will carry.
|
||||
|
||||
See also [users.md](users.md) for what several people share, [configuration.md](configuration.md)
|
||||
for every `[web]` key, and [cli.md](cli.md) for the `ipx user` commands.
|
||||
98
docs/users.md
Normal file
98
docs/users.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Accounts, and what several people share
|
||||
|
||||
ipx serves any number of people from one copy of the data. The rule that decides everything else:
|
||||
**there is one file on disk per enclosure URL.** Two people subscribed to the same show cost one
|
||||
fetch, one parse and one file.
|
||||
|
||||
## What is yours, what is everyone's
|
||||
|
||||
| Yours alone | The same for everyone |
|
||||
|---|---|
|
||||
| Read, kept, playback position | The feed's URL |
|
||||
| Which feeds you see at all | Its download folder |
|
||||
| Keywords, auto-download, explicit, per-scan cap | When it is scanned |
|
||||
| | The file on disk |
|
||||
|
||||
The right-hand column describes the feed and the file rather than a preference — two people wanting
|
||||
different folders would mean two copies. Those three are **admin-only**, and the API returns `403`
|
||||
for anyone else rather than merely hiding the controls.
|
||||
|
||||
## How the scanner merges everyone's wants
|
||||
|
||||
One fetch serves every subscriber, so the policy is a union:
|
||||
|
||||
* an item is downloaded if **anyone** wants it — one person's keyword set matching is enough, and
|
||||
one person with no keywords removes the filter for that feed entirely
|
||||
* auto-download is on if **anyone** has it on
|
||||
* the per-scan cap is the **largest** anyone asked for
|
||||
|
||||
So "auto-download off" means *I don't cause downloads*, not *I never see them*. If someone else's
|
||||
subscription pulls an item, you see it listed as downloaded and can play it, because the enclosure
|
||||
is shared.
|
||||
|
||||
## Deleting
|
||||
|
||||
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
|
||||
for everyone** and names them in the confirmation, and the server has the last word: if anyone else
|
||||
has kept the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
|
||||
reason, and only `?force=true` goes through.
|
||||
|
||||
Retention follows the same rule: an item anyone kept keeps its file, and it counts as read only once
|
||||
every subscriber has read it.
|
||||
|
||||
## Signing in
|
||||
|
||||
Three ways, tried in order of how specific the claim is:
|
||||
|
||||
1. **A proxy header** naming the user — Cloudflare Zero Trust or Authentik. Honoured only from an
|
||||
address in `trusted_proxies`. See [sso.md](sso.md).
|
||||
2. **A session cookie** from signing in at `/login`. Argon2id hashes, sessions in the database,
|
||||
idle timeout `session_days`.
|
||||
3. **The shared `[web] token`**, which signs in as the admin — this is what the Docker healthcheck
|
||||
and any scripts use.
|
||||
|
||||
A database with no accounts creates **admin / ipodderx** on the next daemon start and says so in
|
||||
the log. Change it:
|
||||
|
||||
```sh
|
||||
echo -n 'a good password' | ipx user passwd admin
|
||||
```
|
||||
|
||||
## Adding someone
|
||||
|
||||
```sh
|
||||
echo -n 'their password' | ipx user add sam
|
||||
```
|
||||
|
||||
They sign in at `/login` and start with **no feeds**: subscriptions are per person. Adding a feed
|
||||
someone else already has costs nothing — no second fetch, no second copy — it just appears on their
|
||||
list with their own read state. Unsubscribing removes it from their list alone; only when the last
|
||||
subscriber leaves does the feed stop being scanned, and even then its files and history stay, so
|
||||
re-subscribing does not pull the back catalogue again.
|
||||
|
||||
**Popular** and **Directory** sit at the top of the feed list, above your own feeds. Popular, also
|
||||
shown in the Add feed dialog, lists the ten feeds with the most subscribers on this server, you
|
||||
included. Directory lists every one of them A to Z. Your own feeds are marked Subscribed.
|
||||
It shows a title, artwork and a count, never a URL or who reads it. An OPML subscription is listed
|
||||
as the feeds inside it, one by one, and never the OPML itself, so you can take just the shows you
|
||||
want. Anything that looks private is left out: a login configured for the feed, credentials in its URL,
|
||||
or a key such as `auth=` or `token=` in the query, or a feed from a paid-feed service such as
|
||||
Patreon or Supercast, which put the key in the path, and any feed inside an OPML that looks private
|
||||
itself. Those are someone's paid subscriptions, and listing them would let anyone here read what
|
||||
they pay for.
|
||||
|
||||
An admin can do the same from **Settings → Manage users…**: add someone (with a password, or none
|
||||
for someone the proxy signs in), tick or untick Admin, or remove an account. Removing one takes its
|
||||
subscriptions and read state with it; downloaded files stay. The only admin cannot be demoted or
|
||||
removed there, so there is always someone who can manage the rest.
|
||||
|
||||
## Admin
|
||||
|
||||
The first account is an admin. An admin can change global settings (scanning interval, quota,
|
||||
retention, media types, download folder), a feed's URL, folder and schedule, and who has an account
|
||||
and who else is an admin, and read the log, which names everyone's feeds and sign-ins. Everyone else
|
||||
gets the Settings and Log buttons hidden and a `403` if they ask anyway.
|
||||
|
||||
```sh
|
||||
ipx user list # the admin column says who
|
||||
```
|
||||
58
package-lock.json
generated
Normal file
58
package-lock.json
generated
Normal file
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"name": "ipx-ui-tests",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "ipx-ui-tests",
|
||||
"devDependencies": {
|
||||
"@playwright/test": "^1.56.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@playwright/test": {
|
||||
"version": "1.63.0",
|
||||
"resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.63.0.tgz",
|
||||
"integrity": "sha512-oxMK4vllB9RK5NQ2l1pq1IfOf2AvnEuj/vYGDj0H2nMtmtZpKtCwt/l00GEO6xjGfpBNAvjovvYdCm50dRQkpQ==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"playwright": "1.63.0"
|
||||
},
|
||||
"bin": {
|
||||
"playwright": "cli.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/playwright": {
|
||||
"version": "1.63.0",
|
||||
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.63.0.tgz",
|
||||
"integrity": "sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"playwright-core": "1.63.0"
|
||||
},
|
||||
"bin": {
|
||||
"playwright": "cli.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
},
|
||||
"node_modules/playwright-core": {
|
||||
"version": "1.63.0",
|
||||
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.63.0.tgz",
|
||||
"integrity": "sha512-rYCsBF/M5HjUch52bbtVONEFjv6Xu8sm8h72dNlR5bzIE1fvC/bxgspzkjSfU+MweEMmPM8KJebG6nnyxo5mCg==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"playwright-core": "cli.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
13
package.json
Normal file
13
package.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "ipx-ui-tests",
|
||||
"private": true,
|
||||
"description": "Browser tests for the ipx web UI. The Rust tests cover the server; these cover the page.",
|
||||
"scripts": {
|
||||
"test": "playwright test",
|
||||
"test:headed": "playwright test --headed",
|
||||
"smoke": "node tests/page-smoke.js"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@playwright/test": "^1.56.0"
|
||||
}
|
||||
}
|
||||
43
playwright.config.js
Normal file
43
playwright.config.js
Normal file
@@ -0,0 +1,43 @@
|
||||
const { defineConfig } = require('@playwright/test');
|
||||
const setup = require('./tests/ui/global-setup');
|
||||
|
||||
// Before anything else, including the servers below.
|
||||
setup.prepare();
|
||||
|
||||
// Real browser against a real daemon. The stub-DOM smoke test catches a script that
|
||||
// fails to load; it cannot catch a wrong selector, a handler that runs but does nothing,
|
||||
// or a page that renders empty -- which is exactly what has slipped through before.
|
||||
module.exports = defineConfig({
|
||||
testDir: './tests/ui',
|
||||
timeout: 30_000,
|
||||
expect: { timeout: 10_000 },
|
||||
fullyParallel: false, // one daemon, one database
|
||||
workers: 1,
|
||||
reporter: process.env.CI ? 'line' : [['list']],
|
||||
use: {
|
||||
baseURL: 'http://127.0.0.1:8791',
|
||||
trace: 'retain-on-failure',
|
||||
screenshot: 'only-on-failure',
|
||||
},
|
||||
webServer: [
|
||||
{
|
||||
command: 'node tests/ui/fixtures/serve.js',
|
||||
port: 8792,
|
||||
reuseExistingServer: false,
|
||||
stdout: 'ignore',
|
||||
},
|
||||
{
|
||||
// Build first so the tests always run against current source.
|
||||
command: 'cargo build -q && exec ./target/debug/ipx daemon',
|
||||
port: 8791,
|
||||
reuseExistingServer: false,
|
||||
timeout: 180_000,
|
||||
stdout: 'pipe',
|
||||
env: {
|
||||
IPX_CONFIG: `${setup.root}/config/config.toml`,
|
||||
IPX_DATA_DIR: `${setup.root}/data`,
|
||||
IPX_LOG: 'ipx=info',
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
92
src/auth.rs
Normal file
92
src/auth.rs
Normal file
@@ -0,0 +1,92 @@
|
||||
//! Who is asking. Sign-in is either a local password or a header set by whatever fronts
|
||||
//! this -- Cloudflare Zero Trust on `ipodderx.sdf1.net`, which puts the authenticated
|
||||
//! address in `Cf-Access-Authenticated-User-Email`.
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
use argon2::Argon2;
|
||||
use argon2::password_hash::{PasswordHasher, PasswordVerifier, phc::PasswordHash};
|
||||
|
||||
/// Argon2id with the crate's defaults, which are the OWASP-recommended parameters. The
|
||||
/// salt is generated per password by the hasher itself.
|
||||
pub fn hash_password(password: &str) -> Result<String> {
|
||||
if password.len() < 8 {
|
||||
bail!("password must be at least 8 characters");
|
||||
}
|
||||
Argon2::default()
|
||||
.hash_password(password.as_bytes())
|
||||
.map(|h| h.to_string())
|
||||
.map_err(|e| anyhow::anyhow!("could not hash the password: {e}"))
|
||||
}
|
||||
|
||||
/// False for a wrong password *and* for a stored hash this build cannot parse; either way
|
||||
/// the answer is no.
|
||||
pub fn verify_password(password: &str, stored: &str) -> bool {
|
||||
let Ok(parsed) = PasswordHash::new(stored) else {
|
||||
tracing::warn!("stored password hash is unreadable; refusing the sign-in");
|
||||
return false;
|
||||
};
|
||||
Argon2::default()
|
||||
.verify_password(password.as_bytes(), &parsed)
|
||||
.is_ok()
|
||||
}
|
||||
|
||||
/// A session id: 256 bits of urandom, hex. Long enough that guessing is not a strategy.
|
||||
pub fn new_session_token() -> String {
|
||||
let mut bytes = [0u8; 32];
|
||||
if getrandom(&mut bytes).is_err() {
|
||||
// Falling back to the clock would be a predictable session id. Better to fail.
|
||||
panic!("no source of randomness for a session token");
|
||||
}
|
||||
bytes.iter().map(|b| format!("{b:02x}")).collect()
|
||||
}
|
||||
|
||||
fn getrandom(buf: &mut [u8]) -> std::io::Result<()> {
|
||||
use std::io::Read;
|
||||
std::fs::File::open("/dev/urandom")?.read_exact(buf)
|
||||
}
|
||||
|
||||
/// A username taken from a proxy header. Cloudflare sends an email address; the local part
|
||||
/// is what a person recognises, and the whole thing stays unique enough for one household.
|
||||
pub fn name_from_header(raw: &str) -> Option<String> {
|
||||
let name = raw.trim();
|
||||
if name.is_empty() || name.len() > 190 {
|
||||
return None;
|
||||
}
|
||||
// Anything that could confuse a lookup or a log line is not a name.
|
||||
if name.chars().any(|c| c.is_control() || c == ',' || c == ';') {
|
||||
return None;
|
||||
}
|
||||
Some(name.to_ascii_lowercase())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_password_verifies_only_against_itself() {
|
||||
let h = hash_password("correct horse battery").unwrap();
|
||||
assert!(verify_password("correct horse battery", &h));
|
||||
assert!(!verify_password("Correct horse battery", &h));
|
||||
assert!(!verify_password("", &h));
|
||||
// A hash from a different scheme, or a truncated one, must not authenticate.
|
||||
assert!(!verify_password("correct horse battery", "not-a-hash"));
|
||||
assert!(hash_password("short").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn session_tokens_are_long_and_distinct() {
|
||||
let a = new_session_token();
|
||||
let b = new_session_token();
|
||||
assert_eq!(a.len(), 64);
|
||||
assert_ne!(a, b);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_header_name_is_cleaned_or_refused() {
|
||||
assert_eq!(name_from_header(" Ray@Example.COM "), Some("ray@example.com".into()));
|
||||
assert_eq!(name_from_header(""), None);
|
||||
assert_eq!(name_from_header("ray\nadmin"), None);
|
||||
assert_eq!(name_from_header("ray;admin"), None);
|
||||
}
|
||||
}
|
||||
223
src/config.rs
223
src/config.rs
@@ -5,29 +5,40 @@ use serde::{Deserialize, Serialize};
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
#[derive(Debug, Default, Deserialize, Serialize)]
|
||||
#[derive(Debug, Default, Clone, Deserialize, Serialize)]
|
||||
pub struct Config {
|
||||
#[serde(default)]
|
||||
pub general: General,
|
||||
#[serde(default)]
|
||||
pub torrent: Torrent,
|
||||
#[serde(default)]
|
||||
pub web: Web,
|
||||
/// Keyed by feed id: the TOML table name, which replaces the old genHash(feedURL).
|
||||
#[serde(default)]
|
||||
pub feeds: BTreeMap<String, Feed>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize)]
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
#[serde(default)]
|
||||
pub struct General {
|
||||
pub download_dir: PathBuf,
|
||||
pub socket: PathBuf,
|
||||
/// Default poll interval; a feed's own <ttl> wins when it is longer.
|
||||
pub interval_mins: u64,
|
||||
/// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d".
|
||||
/// A feed's own `schedule` overrides this.
|
||||
pub schedule: String,
|
||||
pub organize: Organize,
|
||||
/// 0 = unlimited.
|
||||
pub max_total_gb: f64,
|
||||
/// 0 = keep forever.
|
||||
pub max_age_days: u64,
|
||||
/// How many new enclosures a single scan may take, when a feed does not say.
|
||||
/// Unlimited by default was a trap: subscribing to an OPML of 80 feeds then pulled
|
||||
/// every back-catalogue episode at once. 0 means unlimited, deliberately chosen.
|
||||
pub max_new_per_check: usize,
|
||||
/// Top-level media types worth downloading. Blog feeds put each article's header
|
||||
/// image in an <enclosure>, so taking everything filled the disk with artwork and
|
||||
/// counted it as episodes. Empty means take anything.
|
||||
pub media_types: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize, Serialize)]
|
||||
@@ -39,7 +50,7 @@ pub enum Organize {
|
||||
Date,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize)]
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
#[serde(default)]
|
||||
pub struct Torrent {
|
||||
pub enabled: bool,
|
||||
@@ -51,6 +62,53 @@ pub struct Torrent {
|
||||
pub stall_mins: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
#[serde(default)]
|
||||
pub struct Web {
|
||||
pub enabled: bool,
|
||||
/// Use 0.0.0.0 to reach it from the LAN. Anything but loopback needs the token.
|
||||
pub bind: String,
|
||||
/// Shared secret. Generated and written back on first run when left empty. It signs
|
||||
/// in as the admin, which is what keeps the healthcheck and any scripts working.
|
||||
pub token: String,
|
||||
/// A header naming the signed-in user, set by whatever fronts this -- Cloudflare Zero
|
||||
/// Trust sends `Cf-Access-Authenticated-User-Email`. Empty disables the whole path.
|
||||
pub trusted_header: String,
|
||||
/// Addresses allowed to assert that header. A header is only as trustworthy as the
|
||||
/// hop that set it, so an empty list means nobody: on a LAN-bound port anyone could
|
||||
/// otherwise claim to be anyone. Loopback covers a tunnel running beside the daemon.
|
||||
pub trusted_proxies: Vec<String>,
|
||||
/// Create an account the first time the proxy vouches for a name it has not seen.
|
||||
pub auto_create_users: bool,
|
||||
/// Where Sign out sends someone the proxy signed in. Signing out of ipx alone cannot stick
|
||||
/// while the proxy still vouches for them, so this is the proxy's own sign-out:
|
||||
/// `/cdn-cgi/access/logout` behind Cloudflare Access. Empty sends them to /login.
|
||||
pub sign_out_url: String,
|
||||
/// Sign a session out after this long without a request.
|
||||
pub session_days: i64,
|
||||
}
|
||||
|
||||
impl Default for Web {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: false,
|
||||
bind: "127.0.0.1:8080".into(),
|
||||
token: String::new(),
|
||||
trusted_header: String::new(),
|
||||
trusted_proxies: vec!["127.0.0.1".into(), "::1".into()],
|
||||
auto_create_users: true,
|
||||
sign_out_url: String::new(),
|
||||
session_days: 30,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Web {
|
||||
pub fn binds_publicly(&self) -> bool {
|
||||
!self.bind.starts_with("127.") && !self.bind.starts_with("localhost")
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
pub struct Feed {
|
||||
pub url: String,
|
||||
@@ -65,7 +123,17 @@ pub struct Feed {
|
||||
pub allow_explicit: bool,
|
||||
#[serde(default = "yes")]
|
||||
pub auto_download: bool,
|
||||
/// Cap on new downloads per scan. None = unlimited.
|
||||
/// Set on feeds that came from a subscribed OPML: the id of the OPML feed they
|
||||
/// belong to. The OPML is re-read on every scan and this list kept in step.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub group: Option<String>,
|
||||
/// Overrides the global schedule for this feed. Same forms: "every 6h", "2d".
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub schedule: Option<String>,
|
||||
/// Media types for this feed. None follows `[general]`; an empty list takes anything.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub media_types: Option<Vec<String>>,
|
||||
/// Cap on new downloads per scan for this feed. None follows `[general]`.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub max_new_per_check: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
@@ -86,10 +154,12 @@ impl Default for General {
|
||||
Self {
|
||||
download_dir: home().join("Podcasts"),
|
||||
socket: default_socket(),
|
||||
interval_mins: 60,
|
||||
schedule: "every 60m".into(),
|
||||
organize: Organize::Feed,
|
||||
max_total_gb: 0.0,
|
||||
max_age_days: 0,
|
||||
max_new_per_check: 3,
|
||||
media_types: vec!["audio".into(), "video".into()],
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -106,6 +176,49 @@ impl Default for Torrent {
|
||||
}
|
||||
}
|
||||
|
||||
impl General {
|
||||
/// Minutes between checks, or an hour when `schedule` is empty or unreadable. A malformed
|
||||
/// value warns rather than stopping the daemon.
|
||||
pub fn interval(&self) -> u64 {
|
||||
if let Some(n) = parse_interval(&self.schedule) {
|
||||
return n;
|
||||
}
|
||||
if !self.schedule.trim().is_empty() {
|
||||
tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default");
|
||||
}
|
||||
60
|
||||
}
|
||||
}
|
||||
|
||||
/// Parses a check interval into minutes.
|
||||
///
|
||||
/// Accepts "every 30m", "30m", "4h", "1d", "2w", "every 4 hours", or a bare number of
|
||||
/// minutes.
|
||||
/// Returns None for anything it cannot read, or for zero.
|
||||
pub fn parse_interval(s: &str) -> Option<u64> {
|
||||
let s = s.trim().to_lowercase();
|
||||
let s = s.strip_prefix("every").unwrap_or(&s).trim();
|
||||
if s.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let digits: String = s.chars().take_while(|c| c.is_ascii_digit()).collect();
|
||||
if digits.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let n: u64 = digits.parse().ok()?;
|
||||
let unit = s[digits.len()..].trim();
|
||||
|
||||
let mins = match unit {
|
||||
"" | "m" | "min" | "mins" | "minute" | "minutes" => n,
|
||||
"h" | "hr" | "hrs" | "hour" | "hours" => n.checked_mul(60)?,
|
||||
"d" | "day" | "days" => n.checked_mul(1440)?,
|
||||
"w" | "week" | "weeks" => n.checked_mul(10080)?,
|
||||
_ => return None,
|
||||
};
|
||||
(mins > 0).then_some(mins)
|
||||
}
|
||||
|
||||
impl Torrent {
|
||||
/// Inclusive listen port range. Falls back to the BitTorrent default on garbage input.
|
||||
pub fn ports(&self) -> (u16, u16) {
|
||||
@@ -166,9 +279,7 @@ pub fn config_path() -> PathBuf {
|
||||
if let Ok(p) = std::env::var("IPX_CONFIG") {
|
||||
return PathBuf::from(p);
|
||||
}
|
||||
dirs::config_dir()
|
||||
.unwrap_or_else(|| home().join(".config"))
|
||||
.join("ipx/config.toml")
|
||||
xdg("XDG_CONFIG_HOME", ".config").join("ipx/config.toml")
|
||||
}
|
||||
|
||||
/// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`.
|
||||
@@ -176,9 +287,7 @@ pub fn data_dir() -> PathBuf {
|
||||
if let Ok(p) = std::env::var("IPX_DATA_DIR") {
|
||||
return PathBuf::from(p);
|
||||
}
|
||||
dirs::data_dir()
|
||||
.unwrap_or_else(|| home().join(".local/share"))
|
||||
.join("ipx")
|
||||
xdg("XDG_DATA_HOME", ".local/share").join("ipx")
|
||||
}
|
||||
|
||||
fn default_socket() -> PathBuf {
|
||||
@@ -188,6 +297,25 @@ fn default_socket() -> PathBuf {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether an enclosure's type is one we want.
|
||||
///
|
||||
/// An unknown type is allowed: the real type is only known after downloading, and
|
||||
/// refusing everything untyped would drop feeds that simply omit the attribute.
|
||||
pub fn wanted_media(mime: Option<&str>, wanted: &[String]) -> bool {
|
||||
if wanted.is_empty() {
|
||||
return true;
|
||||
}
|
||||
let Some(mime) = mime.map(str::trim).filter(|m| !m.is_empty()) else {
|
||||
return true;
|
||||
};
|
||||
let top = mime.split('/').next().unwrap_or(mime).to_ascii_lowercase();
|
||||
// A .torrent is a container for media, not media itself; judge it once unpacked.
|
||||
if mime.to_ascii_lowercase().contains("torrent") {
|
||||
return true;
|
||||
}
|
||||
wanted.iter().any(|w| w.trim().eq_ignore_ascii_case(&top) || w.trim().eq_ignore_ascii_case(mime))
|
||||
}
|
||||
|
||||
/// Feed ids are the TOML table key, so they must be readable and punctuation-free.
|
||||
pub fn slug(text: &str) -> String {
|
||||
let mut out = String::new();
|
||||
@@ -215,8 +343,16 @@ pub fn unique_slug(text: &str, taken: &BTreeMap<String, Feed>) -> String {
|
||||
(2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap()
|
||||
}
|
||||
|
||||
/// `$var`, or `~/fallback` when it is unset or empty, as the XDG base directory spec says.
|
||||
fn xdg(var: &str, fallback: &str) -> PathBuf {
|
||||
std::env::var_os(var)
|
||||
.filter(|v| !v.is_empty())
|
||||
.map(PathBuf::from)
|
||||
.unwrap_or_else(|| home().join(fallback))
|
||||
}
|
||||
|
||||
fn home() -> PathBuf {
|
||||
dirs::home_dir().unwrap_or_else(|| PathBuf::from("."))
|
||||
std::env::var_os("HOME").map(PathBuf::from).unwrap_or_else(|| PathBuf::from("."))
|
||||
}
|
||||
|
||||
fn expand_tilde(p: &Path) -> PathBuf {
|
||||
@@ -236,6 +372,8 @@ mod tests {
|
||||
r#"
|
||||
[general]
|
||||
download_dir = "/tmp/pods"
|
||||
# A key older versions read. An old config that still has it has to load.
|
||||
interval_mins = 45
|
||||
|
||||
[feeds.example]
|
||||
url = "https://example.com/feed.xml"
|
||||
@@ -245,7 +383,7 @@ mod tests {
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(cfg.general.download_dir, PathBuf::from("/tmp/pods"));
|
||||
assert_eq!(cfg.general.interval_mins, 60);
|
||||
assert_eq!(cfg.general.interval(), 60);
|
||||
assert_eq!(cfg.general.organize, Organize::Feed);
|
||||
assert!(cfg.torrent.enabled);
|
||||
|
||||
@@ -256,6 +394,36 @@ mod tests {
|
||||
assert_eq!(feed.keywords, vec!["deep dive"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn intervals_parse_from_the_forms_people_actually_type() {
|
||||
for (input, want) in [
|
||||
("every 30m", 30), ("30m", 30), ("30", 30), ("every 30 minutes", 30),
|
||||
("every 4h", 240), ("4h", 240), ("4 hours", 240), ("EVERY 4H", 240),
|
||||
("1d", 1440), ("every 2 days", 2880), (" every 90m ", 90),
|
||||
("1w", 10080), ("every 2 weeks", 20160), ("2 w", 20160),
|
||||
] {
|
||||
assert_eq!(parse_interval(input), Some(want), "{input:?}");
|
||||
}
|
||||
for bad in ["", " ", "every", "soon", "-5m", "0", "0h", "every 0 minutes", "5 fortnights"] {
|
||||
assert_eq!(parse_interval(bad), None, "{bad:?} should not parse");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn interval_falls_back_to_an_hour() {
|
||||
let mut g = General::default();
|
||||
assert_eq!(g.interval(), 60, "the default schedule");
|
||||
|
||||
g.schedule = "every 15m".into();
|
||||
assert_eq!(g.interval(), 15);
|
||||
|
||||
// Empty or garbage must not stop the daemon.
|
||||
g.schedule = String::new();
|
||||
assert_eq!(g.interval(), 60);
|
||||
g.schedule = "whenever".into();
|
||||
assert_eq!(g.interval(), 60);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn port_range_falls_back_when_malformed() {
|
||||
let mut t = Torrent::default();
|
||||
@@ -268,6 +436,26 @@ mod tests {
|
||||
assert_eq!(t.ports(), (6881, 6889), "reversed range is not a range");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn media_types_keep_article_artwork_out() {
|
||||
let want = vec!["audio".to_string(), "video".to_string()];
|
||||
assert!(wanted_media(Some("audio/mpeg"), &want));
|
||||
assert!(wanted_media(Some("audio/mp4"), &want));
|
||||
assert!(wanted_media(Some("video/quicktime"), &want));
|
||||
assert!(!wanted_media(Some("image/jpeg"), &want), "a blog header image is not an episode");
|
||||
assert!(!wanted_media(Some("text/html"), &want));
|
||||
|
||||
// A torrent is a container; what is inside is judged after unpacking.
|
||||
assert!(wanted_media(Some("application/x-bittorrent"), &want));
|
||||
// Unknown type: only discoverable by downloading, so do not refuse it outright.
|
||||
assert!(wanted_media(None, &want));
|
||||
assert!(wanted_media(Some(""), &want));
|
||||
// An empty list means take anything, which is how it behaved before.
|
||||
assert!(wanted_media(Some("image/jpeg"), &[]));
|
||||
// A full type can be named exactly.
|
||||
assert!(wanted_media(Some("image/jpeg"), &["image/jpeg".to_string()]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn slugs_are_readable_and_unique() {
|
||||
assert_eq!(slug("Accidental Tech Podcast"), "accidental-tech-podcast");
|
||||
@@ -279,7 +467,7 @@ mod tests {
|
||||
|
||||
let mut taken = BTreeMap::new();
|
||||
taken.insert("the-daily".to_string(), Feed {
|
||||
url: "u".into(), folder: None, keywords: vec![], allow_explicit: false,
|
||||
url: "u".into(), folder: None, group: None, media_types: None, schedule: None, keywords: vec![], allow_explicit: false,
|
||||
auto_download: true, max_new_per_check: None, username: None,
|
||||
password: None, password_env: None,
|
||||
});
|
||||
@@ -291,6 +479,9 @@ mod tests {
|
||||
let mut f = Feed {
|
||||
url: "https://x/y".into(),
|
||||
folder: None,
|
||||
group: None,
|
||||
media_types: None,
|
||||
schedule: None,
|
||||
keywords: vec![],
|
||||
allow_explicit: false,
|
||||
auto_download: true,
|
||||
|
||||
105
src/download.rs
105
src/download.rs
@@ -7,19 +7,56 @@ use tokio::io::AsyncWriteExt;
|
||||
|
||||
use crate::config::{Config, Feed as FeedCfg, Organize};
|
||||
|
||||
/// Characters the original's stringCleaning() stripped, plus the control range and the
|
||||
/// trailing dots/spaces it left in. A real length cap is new -- the Python had none.
|
||||
const FORBIDDEN: &[char] = &['/', '\\', '?', '*', ':', '<', '>', '|', '"', '\''];
|
||||
/// Forbidden characters that were separating words: they become "-" so the words stay
|
||||
/// apart. The original's stringCleaning() deleted them, turning "Show | Series" into
|
||||
/// "Show Series".
|
||||
const SEPARATORS: &[char] = &['/', '\\', '|', ':'];
|
||||
/// Forbidden characters that were never separators: they just go.
|
||||
const STRIPPED: &[char] = &['?', '*', '<', '>', '"', '\''];
|
||||
/// A real length cap is new -- the Python had none.
|
||||
const MAX_NAME_BYTES: usize = 255;
|
||||
|
||||
/// Keeps UTF-8: the original transliterated to ASCII via latin1_to_ascii because 2004
|
||||
/// filesystems demanded it. Ours do not.
|
||||
pub fn sanitize(name: &str) -> String {
|
||||
let mut out: String = name
|
||||
let mapped: String = name
|
||||
.chars()
|
||||
.filter(|c| !c.is_control() && !FORBIDDEN.contains(c))
|
||||
.map(|c| if c.is_control() { ' ' } else { c })
|
||||
.filter(|c| !STRIPPED.contains(c))
|
||||
.map(|c| if SEPARATORS.contains(&c) { '-' } else { c })
|
||||
.collect();
|
||||
out = out.trim().trim_matches('.').trim().to_owned();
|
||||
|
||||
// Collapse each run of dashes and spaces into one thing. A run containing a dash
|
||||
// becomes " - " when it also had whitespace ("Show | Series" -> "Show - Series",
|
||||
// "Ep 12: One" -> "Ep 12 - One") and a bare "-" when it did not ("AC/DC" -> "AC-DC").
|
||||
// A run of plain whitespace collapses to a single space.
|
||||
let mut out = String::with_capacity(mapped.len());
|
||||
let mut chars = mapped.chars().peekable();
|
||||
while let Some(c) = chars.next() {
|
||||
if !(c == '-' || c.is_whitespace()) {
|
||||
out.push(c);
|
||||
continue;
|
||||
}
|
||||
let mut has_dash = c == '-';
|
||||
let mut has_space = c.is_whitespace();
|
||||
while let Some(&next) = chars.peek() {
|
||||
if next == '-' {
|
||||
has_dash = true;
|
||||
} else if next.is_whitespace() {
|
||||
has_space = true;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
chars.next();
|
||||
}
|
||||
match (has_dash, has_space) {
|
||||
(true, true) => out.push_str(" - "),
|
||||
(true, false) => out.push('-'),
|
||||
_ => out.push(' '),
|
||||
}
|
||||
}
|
||||
// Leading/trailing separators and dots are noise, and a leading "-" trips up CLI tools.
|
||||
out = out.trim().trim_matches(|c| c == '.' || c == '-').trim().to_owned();
|
||||
|
||||
if out.len() > MAX_NAME_BYTES {
|
||||
// Truncate on a char boundary, keeping the extension if there is a plausible one.
|
||||
@@ -186,7 +223,7 @@ enum Sniffed {
|
||||
/// 2008 and so always answered 'data'.
|
||||
async fn sniff(path: &Path) -> Result<Sniffed> {
|
||||
let head = read_head(path, 512).await?;
|
||||
if infer::is(&head, "torrent") || head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
|
||||
if head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
|
||||
return Ok(Sniffed::Torrent);
|
||||
}
|
||||
let text = String::from_utf8_lossy(&head);
|
||||
@@ -231,17 +268,27 @@ fn unique_path(dir: &Path, name: &str) -> PathBuf {
|
||||
}
|
||||
|
||||
/// Download folder for a feed: per-feed name, or per-day when organize = "date".
|
||||
///
|
||||
/// A folder may name more than one level ("Subscriptions/Some Show") -- feeds from a
|
||||
/// subscribed OPML nest under it -- so each segment is sanitized separately rather than
|
||||
/// letting the sanitizer eat the separator.
|
||||
pub fn folder_for(cfg: &Config, id: &str, feed_cfg: &FeedCfg, title: Option<&str>) -> String {
|
||||
match cfg.general.organize {
|
||||
Organize::Date => chrono::Local::now().format("%m-%d-%Y").to_string(),
|
||||
Organize::Feed => sanitize(
|
||||
feed_cfg
|
||||
Organize::Feed => {
|
||||
let raw = feed_cfg
|
||||
.folder
|
||||
.as_deref()
|
||||
.or(title)
|
||||
.filter(|s| !s.trim().is_empty())
|
||||
.unwrap_or(id),
|
||||
),
|
||||
.unwrap_or(id);
|
||||
raw.split('/')
|
||||
.map(str::trim)
|
||||
.filter(|seg| !seg.is_empty() && *seg != "." && *seg != "..")
|
||||
.map(sanitize)
|
||||
.collect::<Vec<_>>()
|
||||
.join("/")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -264,12 +311,26 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn sanitize_strips_path_and_control_characters() {
|
||||
assert_eq!(sanitize("../../etc/passwd"), "etcpasswd");
|
||||
assert_eq!(sanitize("Ep 12: The \"Best\" One?"), "Ep 12 The Best One");
|
||||
assert_eq!(sanitize("bad\u{0}name\u{7}.mp3"), "badname.mp3");
|
||||
assert_eq!(sanitize("../../etc/passwd"), "etc-passwd");
|
||||
assert_eq!(sanitize("Ep 12: The \"Best\" One?"), "Ep 12 - The Best One");
|
||||
assert_eq!(sanitize("bad\u{0}name\u{7}.mp3"), "bad name .mp3");
|
||||
assert_eq!(sanitize(" spaced.mp3 "), "spaced.mp3");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_turns_separators_into_dashes() {
|
||||
// A real Patreon feed title; the pipes are forbidden characters.
|
||||
assert_eq!(
|
||||
sanitize("Get in the Trunk | Anthology Series | Delta Green"),
|
||||
"Get in the Trunk - Anthology Series - Delta Green"
|
||||
);
|
||||
assert_eq!(sanitize("Ep 12: The One"), "Ep 12 - The One");
|
||||
assert_eq!(sanitize("a b"), "a b", "plain whitespace stays whitespace");
|
||||
assert_eq!(sanitize("AC/DC"), "AC-DC", "no spaces around it, so no spaces added");
|
||||
assert_eq!(sanitize("well-known.mp3"), "well-known.mp3", "existing dashes survive");
|
||||
assert_eq!(sanitize("Show -- Thing"), "Show - Thing");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_never_yields_an_empty_or_dot_name() {
|
||||
assert_eq!(sanitize(""), "download");
|
||||
@@ -319,6 +380,22 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_folder_can_nest_without_the_sanitizer_eating_the_separator() {
|
||||
let mut cfg = Config::default();
|
||||
cfg.general.download_dir = "/tmp".into();
|
||||
let mut f = crate::config::Feed {
|
||||
url: "u".into(), folder: Some("Subscriptions/Some | Show".into()), group: None, media_types: None,
|
||||
schedule: None, keywords: vec![], allow_explicit: false, auto_download: true,
|
||||
max_new_per_check: None, username: None, password: None, password_env: None,
|
||||
};
|
||||
assert_eq!(folder_for(&cfg, "id", &f, None), "Subscriptions/Some - Show");
|
||||
|
||||
// A traversal in a folder name must not climb out of the download directory.
|
||||
f.folder = Some("../../etc/Show".into());
|
||||
assert_eq!(folder_for(&cfg, "id", &f, None), "etc/Show");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn keyword_matching_is_or_across_keywords_and_and_within_one() {
|
||||
let kws = vec!["deep dive".to_string(), "interview".to_string()];
|
||||
|
||||
704
src/feed.rs
704
src/feed.rs
@@ -10,6 +10,7 @@ use crate::config::Feed as FeedCfg;
|
||||
pub struct ParsedFeed {
|
||||
pub title: Option<String>,
|
||||
pub ttl_mins: Option<u64>,
|
||||
pub image: Option<String>,
|
||||
pub entries: Vec<Entry>,
|
||||
}
|
||||
|
||||
@@ -22,10 +23,16 @@ pub struct Entry {
|
||||
pub description: Option<String>,
|
||||
pub categories: Vec<String>,
|
||||
pub explicit: bool,
|
||||
/// Episode artwork; falls back to the feed's in the UI.
|
||||
pub image: Option<String>,
|
||||
/// Seconds.
|
||||
pub duration: Option<i64>,
|
||||
pub episode: Option<i64>,
|
||||
pub season: Option<i64>,
|
||||
pub enclosures: Vec<Enclosure>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default, PartialEq)]
|
||||
#[derive(Debug, Default, Clone, PartialEq)]
|
||||
pub struct Enclosure {
|
||||
pub url: String,
|
||||
pub mime: Option<String>,
|
||||
@@ -80,18 +87,359 @@ pub async fn fetch(
|
||||
Ok(Fetched::Body { bytes, etag, last_modified })
|
||||
}
|
||||
|
||||
/// A stored `last_error`, translated into plain words for whoever subscribes: whose problem
|
||||
/// it is, and whether there is a new address to switch to.
|
||||
pub struct Failure {
|
||||
pub reason: &'static str,
|
||||
pub new_url: Option<String>,
|
||||
}
|
||||
|
||||
/// Reads a `last_error` the same way `set_feed_error` received it (`format!("{e:#}")` on the
|
||||
/// anyhow chain from `fetch` or `parse`) and says what it means, for the errors worth telling
|
||||
/// someone about. Everything else -- a timeout, a 5xx, a 429, a feed that is simply garbled --
|
||||
/// comes back `None`: transient by nature, or with nothing more useful to say than the raw
|
||||
/// text already shown once a feed is open.
|
||||
///
|
||||
/// ponytail: matches on the fixed strings this crate itself produces (`anyhow!("HTTP
|
||||
/// {status}")`, and the "got a web page" message above) plus the substrings a DNS failure
|
||||
/// reliably contains. Fragile if reqwest's own wording changes; the fallback is just showing
|
||||
/// nothing extra, so a miss costs a clearer message, not a wrong one.
|
||||
pub fn explain_failure(msg: &str) -> Option<Failure> {
|
||||
if let Some(rest) = msg.strip_prefix("got a web page, not a feed") {
|
||||
let new_url = rest
|
||||
.strip_prefix("; it links ")
|
||||
.and_then(|r| r.strip_suffix(" as its feed"))
|
||||
.map(str::to_owned);
|
||||
return Some(Failure { reason: "The feed moved; this address now shows a web page.", new_url });
|
||||
}
|
||||
let low = msg.to_ascii_lowercase();
|
||||
if low.contains("http 404") {
|
||||
return Some(Failure { reason: "The publisher took this feed down, or moved it.", new_url: None });
|
||||
}
|
||||
if low.contains("http 401") || low.contains("http 403") {
|
||||
return Some(Failure { reason: "The site refuses ipx's requests.", new_url: None });
|
||||
}
|
||||
if low.contains("http 402") {
|
||||
return Some(Failure { reason: "The feed now needs a paid plan.", new_url: None });
|
||||
}
|
||||
if low.contains("dns error")
|
||||
|| low.contains("failed to lookup address")
|
||||
|| low.contains("no address associated")
|
||||
{
|
||||
return Some(Failure { reason: "This address no longer resolves; the site is gone.", new_url: None });
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// True when a body is an OPML document rather than a feed.
|
||||
///
|
||||
/// The original matched on the URL ending in ".opml" (iPXClass.py:34), which misses an
|
||||
/// OPML served from a URL without that extension. Sniffing the body catches both.
|
||||
pub fn is_opml(bytes: &[u8]) -> bool {
|
||||
let head = &bytes[..bytes.len().min(1024)];
|
||||
let text = String::from_utf8_lossy(head).to_lowercase();
|
||||
text.contains("<opml")
|
||||
}
|
||||
|
||||
/// The feeds listed in an OPML document, as (title, xml_url), walking nested folders.
|
||||
pub fn parse_opml(bytes: &[u8]) -> Result<Vec<(String, String)>> {
|
||||
let text = String::from_utf8_lossy(bytes);
|
||||
let doc = opml::OPML::from_str(&text)
|
||||
.map_err(|e| anyhow!("that does not parse as OPML: {e}"))?;
|
||||
let mut out = vec![];
|
||||
crate::collect_outlines(&doc.body.outlines, &mut out);
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// The <head><title> of an OPML document.
|
||||
pub fn opml_title(bytes: &[u8]) -> Option<String> {
|
||||
let text = String::from_utf8_lossy(bytes);
|
||||
let doc = opml::OPML::from_str(&text).ok()?;
|
||||
doc.head
|
||||
.and_then(|h| h.title)
|
||||
.map(|t| t.trim().to_owned())
|
||||
.filter(|t| !t.is_empty())
|
||||
}
|
||||
|
||||
/// The token and show of a Patreon feed link, or None for any other URL.
|
||||
///
|
||||
/// Patreon gives each patron one token per creator. With no show it stands for the creator,
|
||||
/// whose feed carries every show at once.
|
||||
fn patreon_parts(url: &str) -> Option<(String, Option<String>)> {
|
||||
let u = url::Url::parse(url).ok()?;
|
||||
if !matches!(u.host_str()?, "patreon.com" | "www.patreon.com") || !u.path().starts_with("/rss") {
|
||||
return None;
|
||||
}
|
||||
let param = |name: &str| u.query_pairs().find(|(k, _)| k == name).map(|(_, v)| v.into_owned());
|
||||
Some((param("auth")?, param("show")))
|
||||
}
|
||||
|
||||
/// A Patreon link naming a creator but no show.
|
||||
pub fn is_patreon_creator(url: &str) -> bool {
|
||||
matches!(patreon_parts(url), Some((_, None)))
|
||||
}
|
||||
|
||||
/// What was typed into Add feed, as a URL. A bare Patreon token is taken as its creator's
|
||||
/// feed, since the token alone says whose it is.
|
||||
pub fn expand_input(input: &str) -> String {
|
||||
let s = input.trim();
|
||||
let token = s.len() >= 20 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_');
|
||||
if token { format!("https://www.patreon.com/rss?auth={s}") } else { s.to_owned() }
|
||||
}
|
||||
|
||||
/// Whether two URLs are the same feed. One Patreon show has several spellings -- by the
|
||||
/// creator's name, by number, or with no creator at all -- and the token and show are what
|
||||
/// identify it.
|
||||
pub fn same_feed(a: &str, b: &str) -> bool {
|
||||
a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b))
|
||||
}
|
||||
|
||||
/// A Patreon creator's name and shows, each show as (title, feed URL).
|
||||
///
|
||||
/// ponytail: Patreon's own web API, undocumented, asked without signing in. If it changes,
|
||||
/// finding shows stops and the show feeds already found keep working. The documented API
|
||||
/// needs an OAuth client per install and does not list shows.
|
||||
pub async fn patreon_shows(
|
||||
client: &reqwest::Client,
|
||||
url: &str,
|
||||
) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
|
||||
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
|
||||
let mut resp = client.get(url).send().await.context("connecting")?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
|
||||
}
|
||||
let mut head = Vec::new();
|
||||
while patreon_campaign(&head).is_none() && head.len() < 64 * 1024 {
|
||||
let Some(chunk) = resp.chunk().await.context("reading the feed")? else { break };
|
||||
head.extend_from_slice(&chunk);
|
||||
}
|
||||
let campaign = patreon_campaign(&head)
|
||||
.ok_or_else(|| anyhow!("the Patreon feed does not say whose it is"))?;
|
||||
|
||||
let api = format!(
|
||||
"https://www.patreon.com/api/campaigns/{campaign}\
|
||||
?include=shows&fields%5Bcampaign%5D=name&fields%5Bcollection%5D=title"
|
||||
);
|
||||
let resp = client.get(api).send().await.context("asking Patreon for the shows")?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(anyhow!("Patreon would not list the shows: HTTP {}", resp.status()));
|
||||
}
|
||||
let (name, shows) = parse_patreon_shows(&resp.bytes().await.context("reading the shows")?)?;
|
||||
Ok((name, shows.into_iter().map(|(id, title)| (title, format!("{url}&show={id}"))).collect()))
|
||||
}
|
||||
|
||||
/// The campaign number in the start of a Patreon feed.
|
||||
fn patreon_campaign(head: &[u8]) -> Option<String> {
|
||||
let text = String::from_utf8_lossy(head);
|
||||
text.match_indices("patreon.com/rss/").find_map(|(i, m)| {
|
||||
let id: String = text[i + m.len()..].chars().take_while(char::is_ascii_digit).collect();
|
||||
(!id.is_empty()).then_some(id)
|
||||
})
|
||||
}
|
||||
|
||||
/// A campaign's name and its shows as (id, title), from Patreon's JSON:API answer.
|
||||
fn parse_patreon_shows(json: &[u8]) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||
let v: serde_json::Value = serde_json::from_slice(json).context("Patreon's answer is not JSON")?;
|
||||
// Missing is not the same as none. Read as no shows, the creator feed would be scanned as
|
||||
// a plain feed, claim every show's files, and leave the shows empty once the list returned.
|
||||
let ids = v["data"]["relationships"]["shows"]["data"]
|
||||
.as_array()
|
||||
.ok_or_else(|| anyhow!("Patreon's answer does not list the shows"))?;
|
||||
let title = |id: &str| -> Option<String> {
|
||||
let show = v["included"].as_array()?.iter().find(|x| x["type"] == "collection" && x["id"] == id)?;
|
||||
show["attributes"]["title"].as_str().map(|t| t.trim().to_owned())
|
||||
};
|
||||
let shows = ids
|
||||
.iter()
|
||||
.filter_map(|s| s["id"].as_str())
|
||||
.map(|id| (id.to_owned(), title(id).unwrap_or_else(|| format!("Show {id}"))))
|
||||
.collect();
|
||||
Ok((v["data"]["attributes"]["name"].as_str().map(str::to_owned), shows))
|
||||
}
|
||||
|
||||
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
|
||||
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
|
||||
match rss::Channel::read_from(bytes) {
|
||||
Ok(ch) => Ok(from_rss(ch)),
|
||||
Ok(ch) => Ok(from_rss(ch, bytes)),
|
||||
Err(rss_err) => match atom_syndication::Feed::read_from(bytes) {
|
||||
Ok(feed) => Ok(from_atom(feed)),
|
||||
Err(atom_err) => Err(anyhow!("not RSS ({rss_err}) and not Atom ({atom_err})")),
|
||||
Err(atom_err) => {
|
||||
// Some publishers (kcpw, feedland) write a bare "&" in a URL instead of
|
||||
// "&". Strict XML parsers refuse it; browsers don't. Retry once with
|
||||
// every offending "&" escaped rather than fail outright.
|
||||
let escaped = escape_bare_ampersands(bytes);
|
||||
if escaped != bytes {
|
||||
if let Ok(ch) = rss::Channel::read_from(escaped.as_slice()) {
|
||||
return Ok(from_rss(ch, &escaped));
|
||||
}
|
||||
if let Ok(feed) = atom_syndication::Feed::read_from(escaped.as_slice()) {
|
||||
return Ok(from_atom(feed));
|
||||
}
|
||||
}
|
||||
Err(match alternate_feed_link(bytes) {
|
||||
Some(href) if looks_like_html(bytes) => {
|
||||
anyhow!("got a web page, not a feed; it links {href} as its feed")
|
||||
}
|
||||
None if looks_like_html(bytes) => anyhow!("got a web page, not a feed"),
|
||||
_ => anyhow!("not RSS ({rss_err}) and not Atom ({atom_err})"),
|
||||
})
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn from_rss(ch: rss::Channel) -> ParsedFeed {
|
||||
/// Whether a body is a web page rather than a feed: most of the errors traced back to a feed
|
||||
/// that moved or a domain that lapsed, with the old URL now serving the site instead (or a
|
||||
/// redirect to it). `is_opml` already sniffs the other "not actually a feed" case.
|
||||
fn looks_like_html(bytes: &[u8]) -> bool {
|
||||
let head = String::from_utf8_lossy(&bytes[..bytes.len().min(2048)]).to_lowercase();
|
||||
head.contains("<!doctype html") || head.contains("<html")
|
||||
}
|
||||
|
||||
/// The feed a web page names as its own via `<link rel="alternate" type="application/rss+xml"
|
||||
/// href="...">` (or the Atom equivalent) -- how the new address was found for om.co, ms.now,
|
||||
/// Letters of Note, the Daily Dot, Hell Gate, The Frame Lab and Daily Kos.
|
||||
fn alternate_feed_link(bytes: &[u8]) -> Option<String> {
|
||||
let text = String::from_utf8_lossy(bytes);
|
||||
let lower = text.to_lowercase();
|
||||
let mut pos = 0;
|
||||
while let Some(rel) = lower[pos..].find("<link") {
|
||||
let start = pos + rel;
|
||||
let Some(end) = lower[start..].find('>').map(|e| start + e) else { break };
|
||||
pos = end + 1;
|
||||
let tag = &text[start..end];
|
||||
let tag_lower = &lower[start..end];
|
||||
let is_alternate = tag_lower.contains("rel=\"alternate\"") || tag_lower.contains("rel='alternate'");
|
||||
let is_feed_type = tag_lower.contains("rss+xml") || tag_lower.contains("atom+xml");
|
||||
if is_alternate && is_feed_type
|
||||
&& let Some(href) = tag_attr(tag, "href")
|
||||
{
|
||||
return Some(href);
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// The value of one attribute in an HTML/XML start tag, however it is quoted.
|
||||
fn tag_attr(tag: &str, name: &str) -> Option<String> {
|
||||
let key = format!("{name}=");
|
||||
let idx = tag.to_lowercase().find(&key)?;
|
||||
let after = &tag[idx + key.len()..];
|
||||
let quote = after.chars().next()?;
|
||||
if quote != '"' && quote != '\'' {
|
||||
return None;
|
||||
}
|
||||
let rest = &after[1..];
|
||||
let close = rest.find(quote)?;
|
||||
Some(rest[..close].trim().to_owned())
|
||||
}
|
||||
|
||||
/// Escapes every `&` that does not already start a recognized XML entity
|
||||
/// (`&`, `<`, `>`, `"`, `'`, or a numeric reference like `'`).
|
||||
fn escape_bare_ampersands(bytes: &[u8]) -> Vec<u8> {
|
||||
fn is_entity_start(rest: &[u8]) -> bool {
|
||||
for named in [&b"amp;"[..], b"lt;", b"gt;", b"quot;", b"apos;"] {
|
||||
if rest.starts_with(named) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
let digits = if rest.starts_with(b"#x") || rest.starts_with(b"#X") {
|
||||
&rest[2..]
|
||||
} else if rest.starts_with(b"#") {
|
||||
&rest[1..]
|
||||
} else {
|
||||
return false;
|
||||
};
|
||||
let len = digits.iter().take_while(|b| b.is_ascii_alphanumeric()).count();
|
||||
len > 0 && digits.get(len) == Some(&b';')
|
||||
}
|
||||
|
||||
let mut out = Vec::with_capacity(bytes.len());
|
||||
let mut i = 0;
|
||||
while i < bytes.len() {
|
||||
if bytes[i] == b'&' && !is_entity_start(&bytes[i + 1..]) {
|
||||
out.extend_from_slice(b"&");
|
||||
} else {
|
||||
out.push(bytes[i]);
|
||||
}
|
||||
i += 1;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Every `<enclosure>` of every `<item>`, in document order.
|
||||
///
|
||||
/// The `rss` crate models an item as having at most one enclosure -- which is what RSS 2.0
|
||||
/// says -- and when a feed carries several it keeps only the *last*, silently losing the
|
||||
/// rest. Feeds do ship several, so read them from the XML directly.
|
||||
fn enclosures_by_item(bytes: &[u8]) -> Vec<Vec<Enclosure>> {
|
||||
use quick_xml::events::Event;
|
||||
|
||||
let mut reader = quick_xml::Reader::from_reader(bytes);
|
||||
reader.config_mut().trim_text(true);
|
||||
let mut buf = Vec::new();
|
||||
let mut out: Vec<Vec<Enclosure>> = Vec::new();
|
||||
let mut current: Option<Vec<Enclosure>> = None;
|
||||
|
||||
let read_enclosure = |e: &quick_xml::events::BytesStart| -> Option<Enclosure> {
|
||||
let (mut url, mut mime, mut length) = (String::new(), None, None);
|
||||
for attr in e.attributes().flatten() {
|
||||
// Values arrive escaped: a feed URL's "&" is "&" in the document.
|
||||
let val = quick_xml::escape::unescape(&attr.value)
|
||||
.map(|v| v.trim().to_string())
|
||||
.unwrap_or_default();
|
||||
match attr.key.local_name().as_ref() {
|
||||
"url" => url = val,
|
||||
"type" => mime = Some(val).filter(|v| !v.is_empty()),
|
||||
"length" => length = val.parse().ok(),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
(!url.is_empty()).then_some(Enclosure { url, mime, length })
|
||||
};
|
||||
|
||||
loop {
|
||||
match reader.read_event_into(&mut buf) {
|
||||
Ok(Event::Start(e)) => match e.name().local_name().as_ref() {
|
||||
"item" => current = Some(Vec::new()),
|
||||
"enclosure" => {
|
||||
if let (Some(list), Some(enc)) = (current.as_mut(), read_enclosure(&e)) {
|
||||
list.push(enc);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
},
|
||||
Ok(Event::Empty(e)) => match e.name().local_name().as_ref() {
|
||||
// <item/> with no children still counts, so the indexes stay aligned.
|
||||
"item" => out.push(Vec::new()),
|
||||
"enclosure" => {
|
||||
if let (Some(list), Some(enc)) = (current.as_mut(), read_enclosure(&e)) {
|
||||
list.push(enc);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
},
|
||||
Ok(Event::End(e)) => {
|
||||
if e.name().local_name().as_ref() == "item"
|
||||
&& let Some(list) = current.take()
|
||||
{
|
||||
out.push(list);
|
||||
}
|
||||
}
|
||||
Ok(Event::Eof) | Err(_) => break,
|
||||
_ => {}
|
||||
}
|
||||
buf.clear();
|
||||
}
|
||||
if let Some(list) = current.take() {
|
||||
out.push(list);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn from_rss(ch: rss::Channel, bytes: &[u8]) -> ParsedFeed {
|
||||
let per_item = enclosures_by_item(bytes);
|
||||
let explicit = ch
|
||||
.itunes_ext()
|
||||
.and_then(|it| it.explicit())
|
||||
@@ -100,9 +448,12 @@ fn from_rss(ch: rss::Channel) -> ParsedFeed {
|
||||
let entries = ch
|
||||
.items()
|
||||
.iter()
|
||||
.filter_map(|item| {
|
||||
let enclosures: Vec<Enclosure> = item
|
||||
.enclosure()
|
||||
.enumerate()
|
||||
.filter_map(|(idx, item)| {
|
||||
// Straight from the XML, so an item with several keeps all of them. Falls
|
||||
// back to the parsed one if the scan and the parser disagree on item count.
|
||||
let enclosures: Vec<Enclosure> = per_item.get(idx).cloned().unwrap_or_else(|| {
|
||||
item.enclosure()
|
||||
.into_iter()
|
||||
.map(|e| Enclosure {
|
||||
url: e.url().trim().to_owned(),
|
||||
@@ -110,7 +461,8 @@ fn from_rss(ch: rss::Channel) -> ParsedFeed {
|
||||
length: e.length().parse().ok(),
|
||||
})
|
||||
.filter(|e| !e.url.is_empty())
|
||||
.collect();
|
||||
.collect()
|
||||
});
|
||||
|
||||
let guid = pick_guid(
|
||||
item.guid().map(|g| g.value()),
|
||||
@@ -123,6 +475,7 @@ fn from_rss(ch: rss::Channel) -> ParsedFeed {
|
||||
.itunes_ext()
|
||||
.and_then(|it| it.explicit())
|
||||
.is_some_and(is_yes);
|
||||
let it = item.itunes_ext();
|
||||
|
||||
Some(Entry {
|
||||
guid,
|
||||
@@ -130,7 +483,7 @@ fn from_rss(ch: rss::Channel) -> ParsedFeed {
|
||||
link: non_empty(item.link()),
|
||||
published: item.pub_date().and_then(parse_date),
|
||||
// Content wins over description, as __getEntries preferred entry.content.
|
||||
description: non_empty(item.content()).or_else(|| non_empty(item.description())),
|
||||
description: body(item.content(), item.description()),
|
||||
categories: item
|
||||
.categories()
|
||||
.iter()
|
||||
@@ -138,6 +491,10 @@ fn from_rss(ch: rss::Channel) -> ParsedFeed {
|
||||
.filter(|c| !c.is_empty() && !c.starts_with("http"))
|
||||
.collect(),
|
||||
explicit: explicit || entry_explicit,
|
||||
image: item_image(item, &enclosures),
|
||||
duration: it.and_then(|i| i.duration()).and_then(parse_duration),
|
||||
episode: it.and_then(|i| i.episode()).and_then(|e| e.trim().parse().ok()),
|
||||
season: it.and_then(|i| i.season()).and_then(|e| e.trim().parse().ok()),
|
||||
enclosures,
|
||||
})
|
||||
})
|
||||
@@ -146,6 +503,12 @@ fn from_rss(ch: rss::Channel) -> ParsedFeed {
|
||||
ParsedFeed {
|
||||
title: non_empty(Some(ch.title())),
|
||||
ttl_mins: ch.ttl().and_then(|t| t.trim().parse().ok()),
|
||||
// itunes:image is the square artwork; <image><url> is the older, often smaller one.
|
||||
image: ch
|
||||
.itunes_ext()
|
||||
.and_then(|i| i.image())
|
||||
.map(str::to_owned)
|
||||
.or_else(|| ch.image().map(|i| i.url().to_owned())),
|
||||
entries,
|
||||
}
|
||||
}
|
||||
@@ -186,13 +549,13 @@ fn from_atom(feed: atom_syndication::Feed) -> ParsedFeed {
|
||||
title: non_empty(Some(e.title().as_str())),
|
||||
link: alt.map(str::to_owned),
|
||||
published: e.published().or(Some(e.updated())).map(|d| d.timestamp()),
|
||||
description: e
|
||||
.content()
|
||||
.and_then(|c| c.value())
|
||||
.or_else(|| e.summary().map(|s| s.as_str()))
|
||||
.map(str::to_owned),
|
||||
description: body(e.content().and_then(|c| c.value()), e.summary().map(|s| s.as_str())),
|
||||
categories: e.categories().iter().map(|c| c.term().to_owned()).collect(),
|
||||
explicit: false,
|
||||
image: None,
|
||||
duration: None,
|
||||
episode: None,
|
||||
season: None,
|
||||
enclosures,
|
||||
})
|
||||
})
|
||||
@@ -201,6 +564,7 @@ fn from_atom(feed: atom_syndication::Feed) -> ParsedFeed {
|
||||
ParsedFeed {
|
||||
title: non_empty(Some(feed.title().as_str())),
|
||||
ttl_mins: None,
|
||||
image: feed.logo().or_else(|| feed.icon()).map(str::to_owned),
|
||||
entries,
|
||||
}
|
||||
}
|
||||
@@ -230,6 +594,74 @@ fn non_empty(s: Option<&str>) -> Option<String> {
|
||||
s.map(str::trim).filter(|s| !s.is_empty()).map(str::to_owned)
|
||||
}
|
||||
|
||||
/// An item's show notes: its full body when that is whole, else its description.
|
||||
///
|
||||
/// libsyn served Daily Meditation Podcast's `content:encoded` cut at the `>` inside a class name
|
||||
/// pasted from a web app (`[&:has([data-writing-block])>*]:pointer-events-auto`), so the body
|
||||
/// began halfway through a tag and the page showed the rest of the tag as text. The same item's
|
||||
/// `description` was whole. With no description to fall back on, a damaged body beats none.
|
||||
fn body(content: Option<&str>, description: Option<&str>) -> Option<String> {
|
||||
non_empty(content)
|
||||
.filter(|c| !starts_mid_tag(c))
|
||||
.or_else(|| non_empty(description))
|
||||
.or_else(|| non_empty(content))
|
||||
}
|
||||
|
||||
/// Text that closes an attribute list (`">`) before any tag has opened is the tail of a tag whose
|
||||
/// start was cut off.
|
||||
fn starts_mid_tag(html: &str) -> bool {
|
||||
html[..html.find('<').unwrap_or(html.len())].contains("\">")
|
||||
}
|
||||
|
||||
/// The picture to show beside an item, in order of how deliberate it is:
|
||||
/// `itunes:image`, then Media RSS `media:thumbnail`, then a `media:content` that is an
|
||||
/// image, and finally an image enclosure -- which is how a blog's article picture arrives
|
||||
/// (Substack puts it there), so those entries get artwork rather than a blank square.
|
||||
fn item_image(item: &rss::Item, enclosures: &[Enclosure]) -> Option<String> {
|
||||
if let Some(url) = item.itunes_ext().and_then(|i| i.image()) {
|
||||
return non_empty(Some(url));
|
||||
}
|
||||
|
||||
let media = item.extensions().get("media");
|
||||
let attr = |name: &str, want_image: bool| -> Option<String> {
|
||||
media?.get(name)?.iter().find_map(|e| {
|
||||
if want_image {
|
||||
// media:content carries anything; only take it when it says it is a picture.
|
||||
let is_image = e.attrs.get("type").is_some_and(|t| t.starts_with("image/"))
|
||||
|| e.attrs.get("medium").is_some_and(|m| m == "image");
|
||||
if !is_image {
|
||||
return None;
|
||||
}
|
||||
}
|
||||
non_empty(e.attrs.get("url").map(String::as_str))
|
||||
})
|
||||
};
|
||||
attr("thumbnail", false)
|
||||
.or_else(|| attr("content", true))
|
||||
.or_else(|| {
|
||||
enclosures
|
||||
.iter()
|
||||
.find(|e| e.mime.as_deref().is_some_and(|m| m.starts_with("image/")))
|
||||
.map(|e| e.url.clone())
|
||||
})
|
||||
}
|
||||
|
||||
/// itunes:duration is either plain seconds ("5649") or a clock ("1:34:09", "23:45").
|
||||
fn parse_duration(s: &str) -> Option<i64> {
|
||||
let s = s.trim();
|
||||
if s.is_empty() {
|
||||
return None;
|
||||
}
|
||||
if !s.contains(':') {
|
||||
return s.parse().ok().filter(|n| *n > 0);
|
||||
}
|
||||
let mut total: i64 = 0;
|
||||
for part in s.split(':') {
|
||||
total = total * 60 + part.trim().parse::<i64>().ok()?;
|
||||
}
|
||||
Some(total).filter(|n| *n > 0)
|
||||
}
|
||||
|
||||
/// RSS pubDate is RFC 2822; some feeds ship RFC 3339 instead.
|
||||
fn parse_date(s: &str) -> Option<i64> {
|
||||
let s = s.trim();
|
||||
@@ -274,6 +706,18 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_body_cut_off_mid_tag_gives_way_to_the_description() {
|
||||
// How libsyn served Daily Meditation Podcast #3477: content:encoded began inside a tag.
|
||||
let cut = r#"*]:pointer-events-auto R6Vx5W_threadScrollVars" dir="auto" data-turn="assistant"> <p>What if</p>"#;
|
||||
let whole = r#"<div class="[&:has([data-writing-block])>*]:pointer-events-auto"><p>What if</p></div>"#;
|
||||
assert_eq!(body(Some(cut), Some(whole)).as_deref(), Some(whole));
|
||||
assert_eq!(body(Some("<p>Notes</p>"), Some("Summary")).as_deref(), Some("<p>Notes</p>"), "a whole body wins");
|
||||
assert_eq!(body(Some("Plain notes, no tags."), Some("Summary")).as_deref(), Some("Plain notes, no tags."));
|
||||
assert_eq!(body(Some(cut), None).as_deref(), Some(cut), "a damaged body beats none");
|
||||
assert_eq!(body(None, Some("Summary")).as_deref(), Some("Summary"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn feed_level_explicit_overrides_entries() {
|
||||
let xml = br#"<?xml version="1.0"?>
|
||||
@@ -290,6 +734,46 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explain_failure_translates_the_errors_the_ui_should_flag() {
|
||||
assert_eq!(
|
||||
explain_failure("HTTP 404 Not Found").unwrap().reason,
|
||||
"The publisher took this feed down, or moved it."
|
||||
);
|
||||
assert_eq!(explain_failure("HTTP 401 Unauthorized").unwrap().reason, "The site refuses ipx's requests.");
|
||||
assert_eq!(explain_failure("HTTP 403 Forbidden").unwrap().reason, "The site refuses ipx's requests.");
|
||||
assert_eq!(explain_failure("HTTP 402 Payment Required").unwrap().reason, "The feed now needs a paid plan.");
|
||||
let dns = explain_failure("connecting: dns error: failed to lookup address information").unwrap();
|
||||
assert_eq!(dns.reason, "This address no longer resolves; the site is gone.");
|
||||
let moved = explain_failure("got a web page, not a feed; it links https://x/feed as its feed").unwrap();
|
||||
assert_eq!(moved.new_url.as_deref(), Some("https://x/feed"));
|
||||
assert!(explain_failure("got a web page, not a feed").unwrap().new_url.is_none());
|
||||
for transient in ["HTTP 500 Internal Server Error", "HTTP 429 Too Many Requests", "operation timed out"] {
|
||||
assert!(explain_failure(transient).is_none(), "{transient} must not be flagged");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_web_page_says_so_and_names_the_feed_it_links() {
|
||||
let html = br#"<!doctype html><html><head>
|
||||
<link rel="alternate" type="application/rss+xml" href="https://x.example/feed">
|
||||
</head><body>not a feed</body></html>"#;
|
||||
let err = parse(html).unwrap_err().to_string();
|
||||
assert_eq!(err, "got a web page, not a feed; it links https://x.example/feed as its feed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_web_page_with_no_feed_link_still_says_so() {
|
||||
let html = b"<!doctype html><html><body>moved</body></html>";
|
||||
assert_eq!(parse(html).unwrap_err().to_string(), "got a web page, not a feed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn garbage_that_is_not_html_gets_the_original_parser_errors() {
|
||||
let err = parse(b"not xml at all").unwrap_err().to_string();
|
||||
assert!(err.starts_with("not RSS ("), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_atom_enclosure_links() {
|
||||
let bytes = include_bytes!("../tests/data/atom.xml");
|
||||
@@ -312,6 +796,197 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_bare_ampersand_in_a_link_is_repaired_and_parsed() {
|
||||
// kcpw.org: <link>https://kcpw.org/?post_type=post&p=125715</link> -- a bare "&"
|
||||
// that strict XML rejects but browsers accept.
|
||||
let xml = br#"<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>X</title><link>https://x</link><description>d</description>
|
||||
<item><title>a</title><guid>g1</guid>
|
||||
<link>https://kcpw.org/?post_type=post&p=125715</link>
|
||||
<enclosure url="https://x/a.mp3?a=1&b=2" length="1" type="audio/mpeg"/></item>
|
||||
</channel></rss>"#;
|
||||
let feed = parse(xml).unwrap();
|
||||
assert_eq!(feed.entries[0].link.as_deref(), Some("https://kcpw.org/?post_type=post&p=125715"));
|
||||
assert_eq!(feed.entries[0].enclosures[0].url, "https://x/a.mp3?a=1&b=2");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn escape_bare_ampersands_leaves_real_entities_alone() {
|
||||
let out = escape_bare_ampersands(b"a&b <x> ' / c&d");
|
||||
assert_eq!(out, b"a&b <x> ' / c&d");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_rss_title_always_wins_and_episode_numbers_stay_metadata() {
|
||||
// Some feeds set a different itunes:title. The displayed title is always the RSS
|
||||
// <title>, verbatim -- separators and all -- and season/episode are stored
|
||||
// alongside it rather than folded into it.
|
||||
let xml = br#"<?xml version="1.0"?>
|
||||
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd">
|
||||
<channel><title>Show</title><link>https://x</link><description>d</description>
|
||||
<item>
|
||||
<title>Music from a Darkened Room | Session Zero</title>
|
||||
<itunes:title>Session Zero</itunes:title>
|
||||
<guid>sz</guid>
|
||||
<itunes:season>8</itunes:season>
|
||||
<itunes:duration>6720</itunes:duration>
|
||||
<enclosure url="https://x/sz.mp3" length="1" type="audio/mpeg"/>
|
||||
</item>
|
||||
<item>
|
||||
<title>Music from a Darkened Room Part 1 | Murphy's Drawer</title>
|
||||
<guid>p1</guid>
|
||||
<itunes:season>8</itunes:season><itunes:episode>1</itunes:episode>
|
||||
<enclosure url="https://x/p1.mp3" length="1" type="audio/mpeg"/>
|
||||
</item>
|
||||
</channel></rss>"#;
|
||||
let feed = parse(xml).unwrap();
|
||||
|
||||
let sz = &feed.entries[0];
|
||||
assert_eq!(
|
||||
sz.title.as_deref(),
|
||||
Some("Music from a Darkened Room | Session Zero"),
|
||||
"itunes:title must not override the RSS title"
|
||||
);
|
||||
assert_eq!(sz.season, Some(8));
|
||||
assert_eq!(sz.episode, None, "a missing episode number stays missing");
|
||||
assert_eq!(sz.duration, Some(6720));
|
||||
|
||||
let p1 = &feed.entries[1];
|
||||
assert_eq!(p1.title.as_deref(), Some("Music from a Darkened Room Part 1 | Murphy's Drawer"));
|
||||
assert_eq!((p1.season, p1.episode), (Some(8), Some(1)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn opml_is_recognised_and_its_feeds_listed() {
|
||||
let xml = br#"<opml version="2.0"><head><title>My Subscriptions</title></head><body>
|
||||
<outline text="Folder">
|
||||
<outline type="rss" text="Alpha" xmlUrl="https://a.example/rss"/>
|
||||
<outline type="rss" text="Beta" xmlUrl="https://b.example/rss"/>
|
||||
</outline>
|
||||
<outline text="Not a feed"/>
|
||||
</body></opml>"#;
|
||||
assert!(is_opml(xml));
|
||||
assert_eq!(opml_title(xml).as_deref(), Some("My Subscriptions"));
|
||||
|
||||
let feeds = parse_opml(xml).unwrap();
|
||||
assert_eq!(feeds.len(), 2, "nested folders are walked, non-feed outlines skipped");
|
||||
assert_eq!(feeds[0], ("Alpha".into(), "https://a.example/rss".into()));
|
||||
|
||||
// A feed must never be mistaken for a subscription list.
|
||||
assert!(!is_opml(include_bytes!("../tests/data/rss2.xml")));
|
||||
assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_patreon_creator_is_a_list_of_its_shows() {
|
||||
let tok = "AbCdEfGhIjKlMnOpQrStUvWxYz012_-9";
|
||||
assert_eq!(expand_input(&format!(" {tok} ")), format!("https://www.patreon.com/rss?auth={tok}"));
|
||||
assert_eq!(expand_input("https://example.com/rss"), "https://example.com/rss");
|
||||
|
||||
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss/glasscannon?auth={tok}")));
|
||||
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss?auth={tok}")));
|
||||
assert!(!is_patreon_creator(&format!("https://www.patreon.com/rss/x?auth={tok}&show=1")), "one show is a feed");
|
||||
assert!(!is_patreon_creator(&format!("https://example.com/rss?auth={tok}")));
|
||||
|
||||
// The show you already have by name is the one a bare token would add by number.
|
||||
assert!(same_feed(
|
||||
&format!("https://www.patreon.com/rss/glasscannon?auth={tok}&show=2073588"),
|
||||
&format!("https://www.patreon.com/rss?auth={tok}&show=2073588"),
|
||||
));
|
||||
assert!(!same_feed(
|
||||
&format!("https://www.patreon.com/rss?auth={tok}&show=1"),
|
||||
&format!("https://www.patreon.com/rss?auth={tok}&show=2"),
|
||||
));
|
||||
|
||||
// The self link carries the campaign by number, whichever spelling was asked for.
|
||||
let head = br#"<rss><channel><link>https://www.patreon.com/glasscannon</link>
|
||||
<atom:link href="https://www.patreon.com/rss/369921?auth=t" rel="self"/>"#;
|
||||
assert_eq!(patreon_campaign(head).as_deref(), Some("369921"));
|
||||
assert_eq!(patreon_campaign(b"<rss><channel><title>T"), None);
|
||||
|
||||
let json = br#"{"data":{"id":"369921","type":"campaign","attributes":{"name":"The Glass Cannon Network"},
|
||||
"relationships":{"shows":{"data":[{"id":"2073588","type":"collection"},{"id":"2073636","type":"collection"}]}}},
|
||||
"included":[{"id":"2073588","type":"collection","attributes":{"title":"Get in the Trunk "}},
|
||||
{"id":"2073636","type":"collection","attributes":{"title":"Shadowdark"}}]}"#;
|
||||
let (name, shows) = parse_patreon_shows(json).unwrap();
|
||||
assert_eq!(name.as_deref(), Some("The Glass Cannon Network"));
|
||||
assert_eq!(shows, [("2073588".into(), "Get in the Trunk".into()), ("2073636".into(), "Shadowdark".into())]);
|
||||
|
||||
// An answer that stops naming the shows is an error, never "this creator has none".
|
||||
assert!(parse_patreon_shows(br#"{"data":{"attributes":{"name":"X"}}}"#).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_item_may_carry_several_enclosures() {
|
||||
// The rss crate keeps only one per item -- the last -- so these come from the XML.
|
||||
let xml = br#"<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>M</title><link>https://x</link><description>d</description>
|
||||
<item><title>Two files</title><guid>m1</guid>
|
||||
<enclosure url="https://x/a.mp3?v=1&t=2" length="111" type="audio/mpeg"/>
|
||||
<enclosure url="https://x/b.mp4" length="222" type="video/mp4"/>
|
||||
</item>
|
||||
<item><title>One file</title><guid>m2</guid>
|
||||
<enclosure url="https://x/c.mp3" length="333" type="audio/mpeg"/></item>
|
||||
<item><title>None</title><guid>m3</guid></item>
|
||||
</channel></rss>"#;
|
||||
let f = parse(xml).unwrap();
|
||||
assert_eq!(f.entries.len(), 3);
|
||||
|
||||
let two = &f.entries[0].enclosures;
|
||||
assert_eq!(two.len(), 2, "both enclosures survive");
|
||||
assert_eq!(
|
||||
two[0].url, "https://x/a.mp3?v=1&t=2",
|
||||
"document order, and the escaped ampersand is decoded"
|
||||
);
|
||||
assert_eq!(two[0].length, Some(111));
|
||||
assert_eq!(two[1].url, "https://x/b.mp4");
|
||||
assert_eq!(two[1].mime.as_deref(), Some("video/mp4"));
|
||||
|
||||
assert_eq!(f.entries[1].enclosures.len(), 1);
|
||||
assert_eq!(f.entries[2].enclosures.len(), 0, "an item may have none");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_items_picture_comes_from_the_most_deliberate_source() {
|
||||
let xml = br#"<?xml version="1.0"?>
|
||||
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
|
||||
xmlns:media="http://search.yahoo.com/mrss/">
|
||||
<channel><title>P</title><link>https://x</link><description>d</description>
|
||||
<item><title>Has itunes</title><guid>a</guid>
|
||||
<itunes:image href="https://x/itunes.jpg"/>
|
||||
<media:thumbnail url="https://x/thumb.jpg"/>
|
||||
<enclosure url="https://x/a.jpg" length="1" type="image/jpeg"/></item>
|
||||
<item><title>Has thumbnail</title><guid>b</guid>
|
||||
<media:thumbnail url="https://x/thumb.jpg"/>
|
||||
<enclosure url="https://x/b.jpg" length="1" type="image/jpeg"/></item>
|
||||
<item><title>Has media content</title><guid>c</guid>
|
||||
<media:content url="https://x/pic.jpg" type="image/jpeg"/>
|
||||
<media:content url="https://x/clip.mp4" type="video/mp4"/></item>
|
||||
<item><title>Only an image enclosure</title><guid>d</guid>
|
||||
<enclosure url="https://x/d.jpg" length="1" type="image/jpeg"/></item>
|
||||
<item><title>Audio only</title><guid>e</guid>
|
||||
<enclosure url="https://x/e.mp3" length="1" type="audio/mpeg"/></item>
|
||||
</channel></rss>"#;
|
||||
let f = parse(xml).unwrap();
|
||||
let img = |i: usize| f.entries[i].image.as_deref();
|
||||
assert_eq!(img(0), Some("https://x/itunes.jpg"), "itunes:image wins");
|
||||
assert_eq!(img(1), Some("https://x/thumb.jpg"), "then media:thumbnail");
|
||||
assert_eq!(img(2), Some("https://x/pic.jpg"), "media:content, and only the image one");
|
||||
assert_eq!(img(3), Some("https://x/d.jpg"), "a blog's article picture arrives as an enclosure");
|
||||
assert_eq!(img(4), None, "audio is not a picture");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn durations_parse_from_seconds_or_a_clock() {
|
||||
assert_eq!(parse_duration("5649"), Some(5649));
|
||||
assert_eq!(parse_duration("23:45"), Some(1425));
|
||||
assert_eq!(parse_duration("1:34:09"), Some(5649));
|
||||
assert_eq!(parse_duration("0"), None, "zero is not a duration");
|
||||
assert_eq!(parse_duration(""), None);
|
||||
assert_eq!(parse_duration("garbage"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_html_masquerading_as_a_feed() {
|
||||
assert!(parse(b"<html><body>nope</body></html>").is_err());
|
||||
@@ -324,3 +999,4 @@ mod tests {
|
||||
assert_eq!(pick_guid(None, None, None, None), None);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
126
src/ipc.rs
126
src/ipc.rs
@@ -17,14 +17,17 @@ pub enum Event {
|
||||
FeedError { feed: String, msg: String },
|
||||
Progress {
|
||||
feed: String,
|
||||
/// Which enclosure this is about. Without it a UI cannot tell one download's
|
||||
/// progress from another's and ends up animating every pending row.
|
||||
enclosure: i64,
|
||||
url: String,
|
||||
file: String,
|
||||
done: u64,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
total: Option<u64>,
|
||||
},
|
||||
DownloadDone { feed: String, url: String, path: String, bytes: u64 },
|
||||
DownloadError { feed: String, url: String, msg: String },
|
||||
DownloadDone { feed: String, enclosure: i64, url: String, path: String, bytes: u64 },
|
||||
DownloadError { feed: String, enclosure: i64, url: String, msg: String },
|
||||
TorrentDeferred { feed: String, url: String },
|
||||
Reaped { path: String, bytes: u64 },
|
||||
/// Terminal: a client that asked for work stops reading here.
|
||||
@@ -59,10 +62,10 @@ impl Event {
|
||||
Event::DownloadDone { path, .. } => format!(" saved {path}"),
|
||||
Event::DownloadError { url, msg, .. } => format!(" failed {url}: {msg}"),
|
||||
Event::Reaped { path, bytes } => {
|
||||
format!("reap {path} ({:.1} MB)", *bytes as f64 / 1_048_576.0)
|
||||
format!("deleted {path} ({:.1} MB)", *bytes as f64 / 1_048_576.0)
|
||||
}
|
||||
Event::ReapDone { files, bytes } => format!(
|
||||
"reaped {files} file(s), {:.1} MB",
|
||||
"deleted {files} old file(s), {:.1} MB",
|
||||
*bytes as f64 / 1_048_576.0
|
||||
),
|
||||
Event::Status { feeds, pending, downloaded } => {
|
||||
@@ -70,9 +73,9 @@ impl Event {
|
||||
}
|
||||
Event::Error { msg } => format!("error: {msg}"),
|
||||
// Noise in a terminal; a UI still gets them on the socket.
|
||||
Event::FeedStart { .. } | Event::TorrentDeferred { .. } | Event::ScanDone { .. } => {
|
||||
return None;
|
||||
}
|
||||
Event::FeedStart { feed } => format!("{feed}: checking"),
|
||||
Event::TorrentDeferred { feed, .. } => format!("{feed}: torrent deferred"),
|
||||
Event::ScanDone { feeds } => format!("scan complete, {feeds} feed(s)"),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -90,6 +93,12 @@ pub enum Command {
|
||||
#[serde(default)]
|
||||
dry_run: bool,
|
||||
},
|
||||
/// Fetch one specific enclosure now, ignoring max_new_per_check and the queue order.
|
||||
/// A scan cannot express "this one, now": it takes the lowest-id pending rows up to
|
||||
/// the per-scan cap, so an explicit request has to bypass both.
|
||||
Download {
|
||||
enclosure: i64,
|
||||
},
|
||||
Status,
|
||||
}
|
||||
|
||||
@@ -110,7 +119,45 @@ impl Emitter {
|
||||
}
|
||||
|
||||
pub fn emit(&self, e: Event) {
|
||||
// Also log it. Scans and downloads travel as events, not tracing calls, so
|
||||
// without this the log view shows only startup and HTTP lines and none of the
|
||||
// work the daemon is actually doing. Progress goes to debug: it fires on every
|
||||
// whole percent and would otherwise crowd everything else out of the buffer.
|
||||
// Level by how much it matters. With 80-odd feeds in an OPML subscription, one
|
||||
// line per feed per tick for "not due yet" would push everything worth reading
|
||||
// out of the buffer within a few minutes.
|
||||
let routine = match &e {
|
||||
Event::Progress { .. } | Event::FeedSkip { .. } | Event::FeedStart { .. } => true,
|
||||
Event::FeedDone { new, downloaded, failed, torrents, .. } => {
|
||||
*new == 0 && *downloaded == 0 && *failed == 0 && *torrents == 0
|
||||
}
|
||||
_ => false,
|
||||
};
|
||||
let bad = matches!(
|
||||
&e,
|
||||
Event::FeedError { .. } | Event::DownloadError { .. } | Event::Error { .. }
|
||||
);
|
||||
if let Some(line) = e.human() {
|
||||
let line = line.trim();
|
||||
if bad {
|
||||
tracing::warn!(target: "ipx::scan", "{line}");
|
||||
} else if routine {
|
||||
tracing::debug!(target: "ipx::scan", "{line}");
|
||||
} else {
|
||||
tracing::info!(target: "ipx::scan", "{line}");
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(tx) = &self.tx {
|
||||
// The outbound half of the protocol, as it goes on the wire. Progress is the
|
||||
// high-volume one, so it sits at debug.
|
||||
if let Ok(json) = serde_json::to_string(&e) {
|
||||
if matches!(e, Event::Progress { .. }) {
|
||||
tracing::debug!(target: "ipx::io", "<- {json}");
|
||||
} else {
|
||||
tracing::info!(target: "ipx::io", "<- {json}");
|
||||
}
|
||||
}
|
||||
// An error here only means nobody is listening yet.
|
||||
let _ = tx.send(e.clone());
|
||||
}
|
||||
@@ -125,11 +172,18 @@ pub async fn daemon_is_live(path: &Path) -> bool {
|
||||
UnixStream::connect(path).await.is_ok()
|
||||
}
|
||||
|
||||
/// Answers `status` for the socket, without the worker. The worker runs one job at a time, and a
|
||||
/// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon
|
||||
/// dead. The answer goes to the client that asked and no one else: broadcast, it ended any
|
||||
/// `ipx fetch` that was watching a scan, since `status` is a terminal event.
|
||||
pub type StatusFn = std::sync::Arc<dyn Fn() -> Event + Send + Sync>;
|
||||
|
||||
/// Accepts connections, feeding commands to `cmds` and events from `events` back out.
|
||||
pub async fn serve(
|
||||
path: PathBuf,
|
||||
events: broadcast::Sender<Event>,
|
||||
cmds: mpsc::Sender<Command>,
|
||||
status: StatusFn,
|
||||
) -> Result<()> {
|
||||
// A socket file left by a crashed daemon would block the bind; a live one was already
|
||||
// rejected by the caller's daemon_is_live() check.
|
||||
@@ -148,8 +202,9 @@ pub async fn serve(
|
||||
let (stream, _) = listener.accept().await?;
|
||||
let rx = events.subscribe();
|
||||
let cmds = cmds.clone();
|
||||
let status = status.clone();
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = handle(stream, rx, cmds).await {
|
||||
if let Err(e) = handle(stream, rx, cmds, status).await {
|
||||
tracing::debug!(error = %e, "client gone");
|
||||
}
|
||||
});
|
||||
@@ -160,12 +215,21 @@ async fn handle(
|
||||
stream: UnixStream,
|
||||
mut rx: broadcast::Receiver<Event>,
|
||||
cmds: mpsc::Sender<Command>,
|
||||
status: StatusFn,
|
||||
) -> Result<()> {
|
||||
let (read, mut write) = stream.into_split();
|
||||
|
||||
// Events out.
|
||||
// Events out: everything broadcast, and the answers meant for this client alone.
|
||||
let (reply, mut replies) = mpsc::channel::<Event>(4);
|
||||
let writer = tokio::spawn(async move {
|
||||
while let Ok(ev) = rx.recv().await {
|
||||
loop {
|
||||
let ev = tokio::select! {
|
||||
Some(ev) = replies.recv() => ev,
|
||||
got = rx.recv() => match got {
|
||||
Ok(ev) => ev,
|
||||
Err(_) => break,
|
||||
},
|
||||
};
|
||||
let mut line = serde_json::to_string(&ev).unwrap_or_default();
|
||||
line.push('\n');
|
||||
if write.write_all(line.as_bytes()).await.is_err() {
|
||||
@@ -182,6 +246,15 @@ async fn handle(
|
||||
continue;
|
||||
}
|
||||
match serde_json::from_str::<Command>(line) {
|
||||
// Answered here, not queued behind whatever the worker is on: see StatusFn.
|
||||
Ok(Command::Status) => {
|
||||
tracing::info!(target: "ipx::io", "-> {line}");
|
||||
let ev = status();
|
||||
if let Ok(json) = serde_json::to_string(&ev) {
|
||||
tracing::info!(target: "ipx::io", "<- {json}");
|
||||
}
|
||||
let _ = reply.send(ev).await;
|
||||
}
|
||||
Ok(cmd) => {
|
||||
if cmds.send(cmd).await.is_err() {
|
||||
break; // Worker is gone; so are we.
|
||||
@@ -234,6 +307,11 @@ mod tests {
|
||||
let got: Command = serde_json::from_str(r#"{"cmd":"reap","dry_run":true}"#).unwrap();
|
||||
assert!(matches!(got, Command::Reap { dry_run: true }));
|
||||
|
||||
// "Download this one now" is its own command precisely because a scan cannot
|
||||
// express it: a scan takes the lowest-id pending rows up to max_new_per_check.
|
||||
let got: Command = serde_json::from_str(r#"{"cmd":"download","enclosure":11}"#).unwrap();
|
||||
assert!(matches!(got, Command::Download { enclosure: 11 }));
|
||||
|
||||
assert!(serde_json::from_str::<Command>(r#"{"cmd":"nope"}"#).is_err());
|
||||
}
|
||||
|
||||
@@ -241,6 +319,7 @@ mod tests {
|
||||
fn events_serialise_to_the_documented_shape() {
|
||||
let ev = Event::Progress {
|
||||
feed: "atp".into(),
|
||||
enclosure: 42,
|
||||
url: "https://x/ep.mp3".into(),
|
||||
file: "ep.mp3".into(),
|
||||
done: 10_485_760,
|
||||
@@ -249,10 +328,12 @@ mod tests {
|
||||
let json = serde_json::to_string(&ev).unwrap();
|
||||
assert!(json.starts_with(r#"{"ev":"progress""#), "got {json}");
|
||||
assert!(json.contains(r#""done":10485760"#));
|
||||
assert!(json.contains(r#""enclosure":42"#), "a UI needs this to target one row");
|
||||
|
||||
// total is omitted rather than null when the server sent no length.
|
||||
let ev = Event::Progress {
|
||||
feed: "a".into(),
|
||||
enclosure: 1,
|
||||
url: "u".into(),
|
||||
file: "f".into(),
|
||||
done: 1,
|
||||
@@ -274,4 +355,29 @@ mod tests {
|
||||
}
|
||||
.is_terminal());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn status_is_answered_while_the_worker_is_busy() {
|
||||
// The queue is full and nobody drains it, as when the worker is deep in a long download:
|
||||
// anything sent to it would wait for ever.
|
||||
let (cmds, _worker) = mpsc::channel::<Command>(1);
|
||||
cmds.send(Command::Reap { dry_run: true }).await.unwrap();
|
||||
let (events, _) = broadcast::channel::<Event>(8);
|
||||
// Another client, watching a scan: it must not be handed someone else's answer, which
|
||||
// would end its session.
|
||||
let mut watcher = events.subscribe();
|
||||
let status: StatusFn = std::sync::Arc::new(|| Event::Status { feeds: 1, pending: 2, downloaded: 3 });
|
||||
let (client, server) = UnixStream::pair().unwrap();
|
||||
tokio::spawn(handle(server, events.subscribe(), cmds, status));
|
||||
|
||||
let (read, mut write) = client.into_split();
|
||||
write.write_all(b"{\"cmd\":\"status\"}\n").await.unwrap();
|
||||
let line = tokio::time::timeout(std::time::Duration::from_secs(2), BufReader::new(read).lines().next_line())
|
||||
.await
|
||||
.expect("status waited behind the worker")
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert!(line.contains(r#""ev":"status""#), "{line}");
|
||||
assert!(watcher.try_recv().is_err(), "the answer went to every client, not just the one asking");
|
||||
}
|
||||
}
|
||||
|
||||
147
src/logbuf.rs
Normal file
147
src/logbuf.rs
Normal file
@@ -0,0 +1,147 @@
|
||||
//! In-process ring buffer of log lines, so the UI can show what the daemon is doing.
|
||||
//!
|
||||
//! Tailing a file would not survive Docker, where logs go to stdout and there is no file
|
||||
//! to read. Capturing inside the tracing pipeline works the same either way.
|
||||
|
||||
use std::collections::VecDeque;
|
||||
use std::sync::{LazyLock, Mutex};
|
||||
use tracing::field::{Field, Visit};
|
||||
use tracing_subscriber::Layer;
|
||||
use tracing_subscriber::layer::Context;
|
||||
|
||||
/// Kept small enough to be cheap to hold and to serialise in one response.
|
||||
const CAPACITY: usize = 5000;
|
||||
|
||||
#[derive(Clone, Debug, serde::Serialize)]
|
||||
pub struct LogLine {
|
||||
/// Monotonic, so a client can ask for "everything after N" without duplicates.
|
||||
pub seq: u64,
|
||||
pub ts: i64,
|
||||
pub level: String,
|
||||
pub target: String,
|
||||
pub msg: String,
|
||||
}
|
||||
|
||||
struct Ring {
|
||||
lines: VecDeque<LogLine>,
|
||||
next_seq: u64,
|
||||
}
|
||||
|
||||
static BUF: LazyLock<Mutex<Ring>> = LazyLock::new(|| {
|
||||
Mutex::new(Ring { lines: VecDeque::with_capacity(CAPACITY), next_seq: 1 })
|
||||
});
|
||||
|
||||
pub fn push(level: &str, target: &str, msg: String) {
|
||||
let mut ring = match BUF.lock() {
|
||||
Ok(r) => r,
|
||||
Err(p) => p.into_inner(), // a poisoned log buffer must not take the process down
|
||||
};
|
||||
let seq = ring.next_seq;
|
||||
ring.next_seq += 1;
|
||||
if ring.lines.len() == CAPACITY {
|
||||
ring.lines.pop_front();
|
||||
}
|
||||
ring.lines.push_back(LogLine {
|
||||
seq,
|
||||
ts: crate::db::now(),
|
||||
level: level.to_owned(),
|
||||
target: target.to_owned(),
|
||||
msg,
|
||||
});
|
||||
}
|
||||
|
||||
/// Lines newer than `after`, oldest first, plus the highest seq now held.
|
||||
pub fn since(after: u64, limit: usize) -> (Vec<LogLine>, u64) {
|
||||
let ring = match BUF.lock() {
|
||||
Ok(r) => r,
|
||||
Err(p) => p.into_inner(),
|
||||
};
|
||||
let latest = ring.next_seq.saturating_sub(1);
|
||||
let mut out: Vec<LogLine> = ring
|
||||
.lines
|
||||
.iter()
|
||||
.filter(|l| l.seq > after)
|
||||
.cloned()
|
||||
.collect();
|
||||
// On a first load (after = 0) the tail is what matters, not the head.
|
||||
if out.len() > limit {
|
||||
out.drain(..out.len() - limit);
|
||||
}
|
||||
(out, latest)
|
||||
}
|
||||
|
||||
/// A tracing layer that mirrors every event into the ring.
|
||||
pub struct RingLayer;
|
||||
|
||||
impl<S: tracing::Subscriber> Layer<S> for RingLayer {
|
||||
fn on_event(&self, event: &tracing::Event<'_>, _ctx: Context<'_, S>) {
|
||||
let mut v = Collect::default();
|
||||
event.record(&mut v);
|
||||
let meta = event.metadata();
|
||||
push(meta.level().as_str(), meta.target(), v.finish());
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct Collect {
|
||||
message: String,
|
||||
fields: Vec<String>,
|
||||
}
|
||||
|
||||
impl Collect {
|
||||
fn finish(self) -> String {
|
||||
if self.fields.is_empty() {
|
||||
self.message
|
||||
} else if self.message.is_empty() {
|
||||
self.fields.join(" ")
|
||||
} else {
|
||||
format!("{} {}", self.message, self.fields.join(" "))
|
||||
}
|
||||
}
|
||||
fn add(&mut self, field: &Field, value: String) {
|
||||
if field.name() == "message" {
|
||||
self.message = value;
|
||||
} else {
|
||||
self.fields.push(format!("{}={}", field.name(), value));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Visit for Collect {
|
||||
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
||||
self.add(field, format!("{value:?}"));
|
||||
}
|
||||
// Numbers and bools reach record_debug through the trait's defaults, which prints them the
|
||||
// same way. A string would print quoted there, hence its own method.
|
||||
fn record_str(&mut self, field: &Field, value: &str) {
|
||||
self.add(field, value.to_owned());
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_ring_drops_oldest_and_keeps_sequence_stable() {
|
||||
for i in 0..(CAPACITY + 50) {
|
||||
push("INFO", "t", format!("line {i}"));
|
||||
}
|
||||
let (all, latest) = since(0, CAPACITY * 2);
|
||||
assert_eq!(all.len(), CAPACITY, "bounded");
|
||||
assert!(latest >= (CAPACITY + 50) as u64);
|
||||
assert!(
|
||||
all.first().unwrap().seq < all.last().unwrap().seq,
|
||||
"oldest first"
|
||||
);
|
||||
|
||||
// "everything after the last one I saw" must return nothing new.
|
||||
let (none, _) = since(latest, 100);
|
||||
assert!(none.is_empty());
|
||||
|
||||
// A first load takes the tail, not the head.
|
||||
let (tail, _) = since(0, 5);
|
||||
assert_eq!(tail.len(), 5);
|
||||
assert_eq!(tail.last().unwrap().seq, latest);
|
||||
}
|
||||
}
|
||||
1264
src/main.rs
1264
src/main.rs
File diff suppressed because it is too large
Load Diff
@@ -150,33 +150,52 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn query_never_offers_flagged_files_and_prefers_read_ones() {
|
||||
fn query_never_offers_a_file_anyone_starred_and_prefers_ones_everyone_read() {
|
||||
// One file serves both subscribers, so it takes both of them to release it.
|
||||
let db = Db::memory().unwrap();
|
||||
db.exec_for_test(
|
||||
"INSERT INTO entries (feed_id, guid, first_seen, read, flagged) VALUES
|
||||
('f', 'keep', 0, 1, 1),
|
||||
('f', 'unread', 0, 0, 0),
|
||||
('f', 'read', 0, 1, 0);
|
||||
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
|
||||
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f');
|
||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
||||
('f', 'keep', 0),
|
||||
('f', 'half', 0),
|
||||
('f', 'unread', 0),
|
||||
('f', 'read', 0);
|
||||
-- Starred by one of the two, so it stays whatever the other thinks.
|
||||
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
|
||||
(1, 'f', 'keep', 1, 1),
|
||||
(2, 'f', 'keep', 1, 0),
|
||||
(1, 'f', 'half', 1, 0),
|
||||
(1, 'f', 'read', 1, 0),
|
||||
(2, 'f', 'read', 1, 0);
|
||||
INSERT INTO enclosures (id, feed_id, guid, url, path, bytes_done, state, downloaded_at) VALUES
|
||||
(1, 'f', 'keep', 'u1', '/tmp/keep', 10, 'done', 10),
|
||||
(2, 'f', 'unread', 'u2', '/tmp/unread', 10, 'done', 20),
|
||||
(3, 'f', 'read', 'u3', '/tmp/read', 10, 'done', 30);",
|
||||
(2, 'f', 'half', 'u2', '/tmp/half', 10, 'done', 20),
|
||||
(3, 'f', 'unread', 'u3', '/tmp/unread', 10, 'done', 30),
|
||||
(4, 'f', 'read', 'u4', '/tmp/read', 10, 'done', 40);",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let got: Vec<i64> = db.reap_candidates().unwrap().iter().map(|c| c.id).collect();
|
||||
assert_eq!(got, vec![3, 2], "flagged excluded; read goes before unread");
|
||||
assert_eq!(
|
||||
got,
|
||||
vec![4, 2, 3],
|
||||
"starred by anyone is never offered; read by everyone goes first, and one \
|
||||
person still having it unread keeps it back with the unread ones"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prune_keeps_entries_that_still_have_a_file() {
|
||||
let db = Db::memory().unwrap();
|
||||
db.exec_for_test(
|
||||
"INSERT INTO entries (feed_id, guid, first_seen, read, flagged) VALUES
|
||||
('f', 'has-file', 100, 1, 0),
|
||||
('f', 'no-file', 100, 1, 0),
|
||||
('f', 'flagged', 100, 1, 1),
|
||||
('f', 'recent', 900, 1, 0);
|
||||
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
|
||||
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',1);
|
||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
||||
('f', 'has-file', 100),
|
||||
('f', 'no-file', 100),
|
||||
('f', 'flagged', 100),
|
||||
('f', 'recent', 900);
|
||||
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
|
||||
(1, 'f', 'has-file', 'u1', '/tmp/x', 'done');",
|
||||
)
|
||||
|
||||
1593
src/web.rs
Normal file
1593
src/web.rs
Normal file
File diff suppressed because it is too large
Load Diff
115
tests/page-smoke.js
Normal file
115
tests/page-smoke.js
Normal file
@@ -0,0 +1,115 @@
|
||||
// Executes web/index.html's script against a stub DOM and fails on anything thrown.
|
||||
//
|
||||
// This exists because a ReferenceError at load once blanked the whole UI: a patch
|
||||
// anchored on a function that no longer existed, so `prefsModal` was referenced but
|
||||
// never defined. `node --check` passes that happily -- it is a parse, not a run --
|
||||
// and every server-side test passed too, because the server was fine.
|
||||
//
|
||||
// node tests/page-smoke.js
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const vm = require('vm');
|
||||
|
||||
const html = fs.readFileSync(path.join(__dirname, '..', 'web', 'index.html'), 'utf8');
|
||||
const script = html.split('<script>')[1].split('</script>')[0];
|
||||
const ids = new Set([...html.matchAll(/id="([^"]+)"/g)].map(m => m[1]));
|
||||
|
||||
const missing = [];
|
||||
const el = (name) => new Proxy({ style: { setProperty(){}, getPropertyValue(){ return ''; } }, dataset: {}, classList: { add(){}, remove(){}, toggle(){}, contains(){ return false; } },
|
||||
value: '', textContent: '', innerHTML: '', hidden: false, children: [], firstElementChild: null,
|
||||
appendChild(){}, removeChild(){}, remove(){}, insertAdjacentHTML(){}, addEventListener(){},
|
||||
setAttribute(){}, getAttribute(){ return null; }, select(){}, setSelectionRange(){}, focus(){},
|
||||
replaceWith(){}, querySelector(){ return el('nested'); }, querySelectorAll(){ return []; },
|
||||
play(){ return Promise.resolve(); }, pause(){}, closest(){ return null; } },
|
||||
{ get: (t, k) => k in t ? t[k] : undefined, set: (t, k, v) => (t[k] = v, true) });
|
||||
|
||||
const document = {
|
||||
querySelector(sel) {
|
||||
if (sel.startsWith('#') && !ids.has(sel.slice(1))) { missing.push(sel); return null; }
|
||||
return el(sel);
|
||||
},
|
||||
querySelectorAll: () => [],
|
||||
createElement: () => el('created'),
|
||||
addEventListener(){}, body: el('body'),
|
||||
documentElement: { dataset: {} },
|
||||
};
|
||||
|
||||
const ctx = {
|
||||
document, console,
|
||||
window: { isSecureContext: false, addEventListener(){} },
|
||||
localStorage: { getItem: () => null, setItem(){}, removeItem(){} },
|
||||
navigator: { clipboard: undefined, sendBeacon(){}, mediaSession: undefined },
|
||||
fetch: (url) => Promise.resolve({
|
||||
ok: true, status: 200, text: () => Promise.resolve(''),
|
||||
json: () => Promise.resolve(
|
||||
String(url).includes('/api/settings')
|
||||
? { schedule: 'every 60m', every_mins: 60, download_dir: '/tmp', max_total_gb: 0, max_age_days: 0 }
|
||||
: String(url).includes('/api/users')
|
||||
? [{ id: 1, name: 'admin', admin: true, password: true }, { id: 2, name: 'sam', admin: false, password: false }]
|
||||
: /\/api\/(popular|directory)/.test(String(url))
|
||||
? [{ id: 'f', title: 'A Feed', image: null, subscribers: 2, subscribed: true },
|
||||
{ id: 'g', title: null, image: null, subscribers: 1, subscribed: false }]
|
||||
: /entries/.test(String(url)) ? { total: 0, entries: [] } : []),
|
||||
}),
|
||||
EventSource: function () { this.close = () => {}; },
|
||||
MediaMetadata: function () {},
|
||||
Blob: function () {},
|
||||
setTimeout, clearTimeout, setInterval, clearInterval,
|
||||
confirm: () => false, prompt: () => null, alert(){},
|
||||
Date, Math, JSON, Object, Array, String, Number, Promise, Error, FormData: function(){},
|
||||
URLSearchParams, encodeURIComponent, decodeURIComponent, parseInt, parseFloat, isNaN,
|
||||
};
|
||||
ctx.globalThis = ctx;
|
||||
ctx.window.location = { href: '' };
|
||||
|
||||
try {
|
||||
vm.createContext(ctx);
|
||||
vm.runInContext(script, ctx, { filename: 'index.html<script>', timeout: 5000 });
|
||||
} catch (e) {
|
||||
console.error('FAIL: the page script threw while loading\n ' + e.stack.split('\n').slice(0, 3).join('\n '));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// The modals are built on demand, so a load-time check never reaches them. Drive the
|
||||
// ones that construct markup from live data, which is where a bad field reference hides.
|
||||
const feed = {
|
||||
id: 'f', url: 'https://x/rss', title: 'A Feed', image: null, folder: null,
|
||||
keywords: ['a'], allow_explicit: false, auto_download: true, max_new_per_check: 3,
|
||||
schedule: 'every 6h', schedule_mins: 360, every_mins: 360,
|
||||
last_checked: 1, next_check: 2, entries: 1, downloaded: 0, unread: 1, last_error: null,
|
||||
};
|
||||
const drive = [
|
||||
['settingsModal', () => ctx.settingsModal(feed)],
|
||||
['settingsModal (no override)', () => ctx.settingsModal({ ...feed, schedule: null, schedule_mins: null })],
|
||||
['downloadLatestModal', () => ctx.downloadLatestModal(feed)],
|
||||
['removeFeed', () => ctx.removeFeed(feed)],
|
||||
['prefsModal', () => ctx.prefsModal()],
|
||||
['usersModal', () => ctx.usersModal()],
|
||||
['opmlModal', () => ctx.opmlModal()],
|
||||
['selectFeed (directory)', () => ctx.selectFeed(':directory')],
|
||||
['selectFeed (popular)', () => ctx.selectFeed(':popular')],
|
||||
['selectFeed (all subscriptions)', () => ctx.selectFeed(':all')],
|
||||
['logsModal', () => ctx.logsModal()],
|
||||
// `const S` is not reachable from here: top-level const/let do not become properties
|
||||
// of a vm context the way var and function declarations do.
|
||||
['renderGroup', () => ctx.renderGroup(feed, [{ ...feed, id: 'child', group: 'f', orphaned: true }])],
|
||||
];
|
||||
for (const [name, fn] of drive) {
|
||||
try {
|
||||
const r = fn();
|
||||
if (r && typeof r.catch === 'function') r.catch(e => {
|
||||
console.error(`FAIL: ${name} rejected: ${e.message}`); process.exit(1);
|
||||
});
|
||||
} catch (e) {
|
||||
console.error(`FAIL: ${name} threw: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (missing.length) {
|
||||
console.error('FAIL: handlers wired to elements that do not exist: ' + [...new Set(missing)].join(', '));
|
||||
process.exit(1);
|
||||
}
|
||||
console.log('OK: page script loads clean, every selector it wires at load exists');
|
||||
// logsModal arms a poll timer; without this the pending interval keeps node alive.
|
||||
process.exit(0);
|
||||
822
tests/ui/app.spec.js
Normal file
822
tests/ui/app.spec.js
Normal file
@@ -0,0 +1,822 @@
|
||||
const { test, expect } = require('@playwright/test');
|
||||
const { TOKEN } = require('./global-setup');
|
||||
|
||||
// The token sets a cookie, so every test starts by presenting it once.
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto(`/?token=${TOKEN}`);
|
||||
await expect(page.locator('#feedlist')).toBeVisible();
|
||||
});
|
||||
|
||||
test('the page loads and lists the configured feeds', async ({ page }) => {
|
||||
// Regression: a ReferenceError in the script left the shell rendered and the sidebar
|
||||
// empty, with every handler below the error dead. Server-side checks all passed.
|
||||
// Four top-level feeds in the fixture config; the OPML's children are inside a closed folder.
|
||||
await expect(page.locator('.feed')).toHaveCount(5, { timeout: 15_000 });
|
||||
await expect(page.getByText('Test Show')).toBeVisible();
|
||||
const errors = [];
|
||||
page.on('pageerror', e => errors.push(e.message));
|
||||
await page.reload();
|
||||
await expect(page.locator('.feed').first()).toBeVisible();
|
||||
expect(errors, 'the page script must not throw at load').toEqual([]);
|
||||
});
|
||||
|
||||
test('the theme toggle actually changes the theme', async ({ page }) => {
|
||||
// Regression: this button was wired after a line that threw, so it did nothing.
|
||||
const before = await page.evaluate(() => document.documentElement.dataset.theme || 'system');
|
||||
await page.locator('#theme').click();
|
||||
await expect
|
||||
.poll(() => page.evaluate(() => document.documentElement.dataset.theme))
|
||||
.not.toBe(before);
|
||||
});
|
||||
|
||||
test('the theme button steps through dark, light and classic, and remembers', async ({ page }) => {
|
||||
const theme = () => page.evaluate(() => document.documentElement.dataset.theme);
|
||||
for (let i = 0; i < 3 && (await theme()) !== 'classic'; i++) await page.locator('#theme').click();
|
||||
expect(await theme()).toBe('classic');
|
||||
await expect(page.locator('#theme')).toHaveAttribute('title', /Classic.*Click for Dark/);
|
||||
|
||||
await page.reload();
|
||||
await expect.poll(theme).toBe('classic');
|
||||
// The 2004 Mac app set its type in Lucida Grande.
|
||||
expect(await page.evaluate(() => getComputedStyle(document.body).fontFamily)).toContain('Lucida Grande');
|
||||
});
|
||||
|
||||
test('settings opens and saves the global schedule', async ({ page }) => {
|
||||
await page.locator('#prefs').click();
|
||||
await expect(page.locator('#modal.on')).toBeVisible();
|
||||
await expect(page.locator('#gnum')).toBeVisible();
|
||||
await page.locator('#gnum').fill('4');
|
||||
await page.locator('#gunit').selectOption('h');
|
||||
await page.locator('#gsave').click();
|
||||
await expect(page.locator('#modal.on')).toBeHidden();
|
||||
|
||||
// It must survive a reload, i.e. actually reach the config.
|
||||
await page.locator('#prefs').click();
|
||||
await expect(page.locator('#gnum')).toHaveValue('4');
|
||||
await expect(page.locator('#gunit')).toHaveValue('h');
|
||||
});
|
||||
|
||||
test('episodes show with their metadata, and the text opens below', async ({ page }) => {
|
||||
await page.getByText('Test Show').click();
|
||||
await expect(page.locator('.ep').first()).toBeVisible({ timeout: 20_000 });
|
||||
await expect(page.getByText('First Episode')).toBeVisible();
|
||||
// Newest first, so target the episode by name rather than by position.
|
||||
const first = page.locator('.ep', { hasText: 'First Episode' });
|
||||
await expect(first).toContainText('S1E1');
|
||||
await expect(first).toContainText('30:30'); // itunes:duration 1830
|
||||
await expect(page.locator('.ep', { hasText: 'Second Episode' })).toContainText('15:00');
|
||||
|
||||
// Selecting an item shows its text in the pane below, not inline in the row.
|
||||
await first.click();
|
||||
await expect(first).toHaveClass(/sel/);
|
||||
await expect(page.locator('#detail')).toContainText('Show notes for the first one');
|
||||
await expect(page.locator('#detail .dt')).toHaveText('First Episode');
|
||||
});
|
||||
|
||||
test('the three panes are there and the item text lands in the bottom one', async ({ page }) => {
|
||||
await page.getByText('Test Show').click();
|
||||
await expect(page.locator('#list')).toBeVisible();
|
||||
await expect(page.locator('#grab')).toBeVisible(); // the draggable divider
|
||||
await expect(page.locator('#detail')).toContainText('Pick an item');
|
||||
|
||||
await page.locator('.ep', { hasText: 'First Episode' }).click();
|
||||
await expect(page.locator('#detail .dt')).toHaveText('First Episode');
|
||||
// The enclosure goes to the Files pane beside the list, as the original's did.
|
||||
await expect(page.locator('#files')).toBeVisible();
|
||||
await expect(page.locator('#files .encbox')).toHaveCount(1);
|
||||
|
||||
// Only the downloaded one gets a player, and max_new_per_check is 1, so find it by
|
||||
// its chip rather than assuming which episode the daemon happened to fetch.
|
||||
const downloaded = page.locator('.ep', { has: page.locator('.kind.here') }).first();
|
||||
await downloaded.click();
|
||||
await expect(page.locator('#files [data-a="play"]')).toBeVisible();
|
||||
await expect(page.locator('#files .encbox [title="Save to this computer"]')).toBeVisible();
|
||||
|
||||
// Selecting another item replaces the pane rather than stacking.
|
||||
await page.locator('.ep', { hasText: 'First Episode' }).click();
|
||||
await expect(page.locator('#detail .dt')).toHaveText('First Episode');
|
||||
await expect(page.locator('#files [data-a="play"]')).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('a downloaded file that is not audio gets no player', async ({ page }) => {
|
||||
// Regression: anything with a file got an <audio> element and a play button, so a blog's
|
||||
// header image rendered as a broken player.
|
||||
await page.locator('.feed', { hasText: 'Picture Blog' }).click();
|
||||
const row = page.locator('.ep', { hasText: 'An Article' });
|
||||
await expect(row).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
await expect(row.locator('[data-a="play"]')).toHaveCount(0);
|
||||
await row.click();
|
||||
|
||||
await expect(page.locator('#detail .dt')).toHaveText('An Article');
|
||||
await expect(page.locator('#files [data-a="play"]')).toHaveCount(0);
|
||||
// What it is and that it is here: one icon, green, with the words in its tooltip.
|
||||
await expect(page.locator('#files .encbox .kind.here')).toHaveAttribute('title', 'image, downloaded');
|
||||
// Still offered as a file, just not as an episode: viewable and keepable.
|
||||
await expect(page.locator('#files [title="Save to this computer"]')).toBeVisible();
|
||||
const view = page.locator('#files a[title="View in a new tab"]');
|
||||
await expect(view).toHaveAttribute('target', '_blank');
|
||||
await expect(view).toHaveAttribute('rel', /noopener/);
|
||||
await expect(view).toHaveAttribute('href', /\/media\/\d+/);
|
||||
});
|
||||
|
||||
test('an item with several enclosures lists them all', async ({ page }) => {
|
||||
await page.locator('.feed', { hasText: 'Multi Show' }).click();
|
||||
const row = page.locator('.ep', { hasText: 'Two Files' });
|
||||
await expect(row).toBeVisible({ timeout: 20_000 });
|
||||
// The row says there is more than one without listing them.
|
||||
await expect(row).toContainText('+1 more file');
|
||||
|
||||
await row.click();
|
||||
// The Files pane lists every one: the audio and the image.
|
||||
await expect(page.locator('#files .encbox')).toHaveCount(2);
|
||||
await expect(page.locator('#files .encbox').nth(1).locator('.kind[title^="image"]')).toBeVisible();
|
||||
});
|
||||
|
||||
test('the filter tabs change what is listed', async ({ page }) => {
|
||||
await page.getByText('Test Show').click();
|
||||
await expect(page.locator('.ep').first()).toBeVisible({ timeout: 20_000 });
|
||||
const all = await page.locator('.ep').count(); // All is the default tab
|
||||
await expect(page.locator('#count')).toContainText('item');
|
||||
|
||||
await page.locator('.tabs button', { hasText: 'Unread' }).first().click();
|
||||
expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all);
|
||||
|
||||
await page.locator('.tabs button', { hasText: 'Kept' }).first().click();
|
||||
await expect(page.locator('#count')).toContainText('0 items');
|
||||
});
|
||||
|
||||
test('a feed URL is editable and has a copy button', async ({ page }) => {
|
||||
await page.getByText('Test Show').click();
|
||||
await page.locator('#content .acts [data-a="settings"]').click();
|
||||
await expect(page.locator('#surl')).toHaveValue(/show\.xml/);
|
||||
await expect(page.locator('#scopy')).toBeVisible();
|
||||
|
||||
// navigator.clipboard is absent over plain http, so the button must not throw.
|
||||
const errors = [];
|
||||
page.on('pageerror', e => errors.push(e.message));
|
||||
await page.locator('#scopy').click();
|
||||
await expect(page.locator('#scopy')).toHaveText(/Copied|Failed/);
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
test('the log view has tabs and shows daemon traffic', async ({ page }) => {
|
||||
await page.locator('#logs').click();
|
||||
await expect(page.locator('#logbox')).toBeVisible();
|
||||
await expect(page.locator('#logtabs button')).toHaveCount(4);
|
||||
|
||||
// Generate traffic, then check the Daemon I/O tab shows both directions.
|
||||
await page.locator('#logtabs button', { hasText: 'Daemon I/O' }).click();
|
||||
await page.evaluate(() =>
|
||||
fetch('/api/fetch', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ force: true }),
|
||||
}));
|
||||
await expect(page.locator('#logbox .l').first()).toBeVisible({ timeout: 15_000 });
|
||||
await expect(page.locator('#logbox')).toContainText('"cmd":"fetch"', { timeout: 15_000 });
|
||||
await expect(page.locator('#logbox')).toContainText('"ev":', { timeout: 15_000 });
|
||||
});
|
||||
|
||||
test('an OPML subscription is a collapsible folder', async ({ page }) => {
|
||||
// Read the subscription so its feeds exist.
|
||||
await page.evaluate(() =>
|
||||
fetch('/api/fetch', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
||||
}));
|
||||
|
||||
// Only a folder has a triangle, and it is a button that says whether the folder is open.
|
||||
const chev = page.locator('.feed.group .chev');
|
||||
await expect(chev).toBeVisible({ timeout: 20_000 });
|
||||
await expect(page.locator('.feed:not(.group) .chev')).toHaveCount(0);
|
||||
await expect(chev).toHaveAttribute('aria-expanded', 'false');
|
||||
|
||||
// Closed by default: the children are not listed until the folder is opened.
|
||||
const before = await page.locator('.feed').count();
|
||||
await chev.click();
|
||||
await expect(page.locator('.feed')).toHaveCount(before + 2);
|
||||
// Scoped to the sidebar: the name also appears as the page heading once selected.
|
||||
await expect(page.locator('#feedlist').getByText('Grouped Show')).toBeVisible();
|
||||
|
||||
// The subscription's own page lists what is inside it.
|
||||
await page.locator('.feed', { hasText: 'Test Subscriptions' }).first().click();
|
||||
await expect(page.locator('.childrow')).toHaveCount(2);
|
||||
});
|
||||
|
||||
test('inside an OPML, feeds with unread items are listed first', async ({ page }) => {
|
||||
await page.evaluate(() =>
|
||||
fetch('/api/fetch', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
||||
}));
|
||||
const chev = page.locator('.feed.group .chev');
|
||||
await expect(chev).toBeVisible({ timeout: 20_000 });
|
||||
await chev.click();
|
||||
const side = page.locator('#feedlist');
|
||||
await expect(side.getByText('Aardvark Radio')).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
// Other tests change read state, so set it here: Aardvark Radio read, Grouped Show not.
|
||||
// Opening an item reads it; the toggle in the pane below flips it back.
|
||||
await side.getByText('Aardvark Radio').click();
|
||||
const aa = page.locator('.ep', { hasText: 'Aardvark Ep' });
|
||||
await aa.click();
|
||||
await expect(aa).toHaveClass(/read/);
|
||||
await side.getByText('Grouped Show').click();
|
||||
const gs = page.locator('.ep', { hasText: 'Grouped Ep' });
|
||||
await gs.click();
|
||||
await page.locator('#detail button[title="Mark unread"]').click();
|
||||
await expect(gs).not.toHaveClass(/read/);
|
||||
|
||||
// Aardvark comes first alphabetically and in the OPML, so only the unread sort puts
|
||||
// Grouped Show above it. The folder stays open across the reload (localStorage).
|
||||
await page.reload();
|
||||
const want = ['Grouped Show', 'Aardvark Radio'];
|
||||
await expect(page.locator('#feedlist .feed.child b')).toHaveText(want, { timeout: 20_000 });
|
||||
await page.locator('.feed', { hasText: 'Test Subscriptions' }).first().click();
|
||||
await expect(page.locator('.childrow b')).toHaveText(want);
|
||||
});
|
||||
|
||||
test('marking an OPML subscription read covers the feeds inside it', async ({ page }) => {
|
||||
await page.evaluate(() =>
|
||||
fetch('/api/fetch', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
||||
}));
|
||||
|
||||
// The folder's own row has no entries, so anything it marks read came from its child.
|
||||
const folder = page.locator('.feed', { hasText: 'Test Subscriptions' }).first();
|
||||
await expect(folder).toBeVisible({ timeout: 20_000 });
|
||||
await expect(folder.locator('.badge')).not.toHaveText('0');
|
||||
await folder.click();
|
||||
|
||||
await page.locator('#content .acts [data-a="read"]').click();
|
||||
await expect(folder.locator('.badge')).toHaveText('0');
|
||||
});
|
||||
|
||||
test.describe('on a phone', () => {
|
||||
test.use({ viewport: { width: 390, height: 844 } });
|
||||
|
||||
test('the feed list is reachable and an item reads full screen', async ({ page }) => {
|
||||
// The burger used to live in the player bar, which is hidden until something plays --
|
||||
// leaving no way to reach the feeds at all.
|
||||
await expect(page.locator('#burger')).toBeVisible();
|
||||
await expect(page.locator('#player')).not.toBeVisible();
|
||||
|
||||
await page.locator('#burger').click();
|
||||
await page.locator('.feed', { hasText: 'Test Show' }).first().click();
|
||||
await expect(page.locator('#sidebar')).not.toHaveClass(/open/);
|
||||
|
||||
// Nothing may push the page sideways at this width.
|
||||
const over = await page.evaluate(() =>
|
||||
document.documentElement.scrollWidth - window.innerWidth);
|
||||
expect(over).toBeLessThanOrEqual(0);
|
||||
|
||||
// One pane at a time: the list, then the item over it, then back.
|
||||
await expect(page.locator('#detail')).not.toBeVisible();
|
||||
await page.locator('.ep').first().click();
|
||||
await expect(page.locator('#detail')).toBeVisible();
|
||||
await page.locator('#dback').click();
|
||||
await expect(page.locator('#detail')).not.toBeVisible();
|
||||
});
|
||||
});
|
||||
|
||||
test('opening an item marks it read, and the toggle flips it back', async ({ page }) => {
|
||||
const errors = [];
|
||||
page.on('pageerror', e => errors.push(e.message));
|
||||
|
||||
await page.getByText('Test Show').click();
|
||||
const row = () => page.locator('.ep', { hasText: 'Second Episode' });
|
||||
await expect(row()).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
// Another test may have opened this item already, so start from a known state: the
|
||||
// toggle in the text below flips it back -- which used to recurse until the stack blew.
|
||||
await row().click();
|
||||
await page.locator('#detail button[title="Mark unread"]').click();
|
||||
await expect(row()).not.toHaveClass(/read/);
|
||||
await expect(page.locator('#detail button[title="Mark read"]')).toBeVisible();
|
||||
|
||||
// Opening it is reading it.
|
||||
await row().click();
|
||||
await expect(row()).toHaveClass(/read/);
|
||||
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
test('the toolbar acts on the selected item', async ({ page }) => {
|
||||
await page.getByText('Test Show').click();
|
||||
const row = () => page.locator('.ep', { hasText: 'Second Episode' });
|
||||
await expect(row()).toBeVisible({ timeout: 20_000 });
|
||||
// Nothing selected, nothing to act on.
|
||||
await expect(page.locator('#tbRead')).toBeDisabled();
|
||||
|
||||
await row().click(); // opening it reads it
|
||||
await expect(row()).toHaveClass(/read/);
|
||||
await page.locator('#tbRead').click();
|
||||
await expect(row()).not.toHaveClass(/read/);
|
||||
|
||||
await page.locator('#tbFlag').click();
|
||||
await expect(row().locator('.fl')).toHaveClass(/on/);
|
||||
await page.locator('#tbFlag').click(); // and back, so later tests see it unkept
|
||||
await expect(row().locator('.fl')).not.toHaveClass(/on/);
|
||||
|
||||
// Second Episode is the one the daemon downloaded, so it plays from the toolbar.
|
||||
await expect(page.locator('#tbPlay')).toBeEnabled();
|
||||
await page.locator('#tbPlay').click();
|
||||
await expect(page.locator('#player')).toBeVisible();
|
||||
await page.locator('#pclose').click();
|
||||
});
|
||||
|
||||
test('a second person has their own feeds and their own read state', async ({ browser }) => {
|
||||
const { execFileSync } = require('child_process');
|
||||
const setup = require('./global-setup');
|
||||
const env = {
|
||||
...process.env,
|
||||
IPX_CONFIG: `${setup.root}/config/config.toml`,
|
||||
IPX_DATA_DIR: `${setup.root}/data`,
|
||||
};
|
||||
try {
|
||||
execFileSync('./target/debug/ipx', ['user', 'add', 'sam'], { input: 'sampassword', env });
|
||||
} catch (e) {
|
||||
if (!String(e.stderr || e.stdout).includes('already exists')) throw e;
|
||||
}
|
||||
|
||||
// A fresh context, so none of the admin's cookies come along.
|
||||
const ctx = await browser.newContext();
|
||||
const page = await ctx.newPage();
|
||||
await page.goto('/login');
|
||||
// The sign-in page shows the icon, so it has to load before anyone has signed in.
|
||||
const icon = await page.request.get('/icon.png');
|
||||
expect(icon.status()).toBe(200);
|
||||
expect(icon.headers()['content-type']).toBe('image/png');
|
||||
await page.locator('#name').fill('sam');
|
||||
await page.locator('#pw').fill('sampassword');
|
||||
await page.locator('button[type=submit]').click();
|
||||
await expect(page.locator('#feedlist')).toBeVisible();
|
||||
|
||||
// Sam subscribes to nothing yet, so sees nothing -- the admin's feeds are not theirs.
|
||||
await expect(page.locator('#feedlist')).toContainText('No feeds.');
|
||||
// Settings stays: Sam has their own subscriptions to export and import, and the
|
||||
// schedule and quota are worth seeing even without a say in them. Only the log and the
|
||||
// users screen -- and the server -- are an admin's alone.
|
||||
await expect(page.locator('#prefs')).toBeVisible();
|
||||
await page.locator('#prefs').click();
|
||||
await expect(page.locator('#modalCard')).toContainText('Subscriptions');
|
||||
await expect(page.locator('#gsave')).toBeHidden();
|
||||
await expect(page.locator('#gusers')).toBeHidden();
|
||||
await page.locator('#modalCard .cardacts .btn').first().click();
|
||||
// Hiding the button is not the guard; the server is.
|
||||
expect((await page.request.get('/api/users')).status()).toBe(403);
|
||||
await expect(page.locator('#logs')).toBeHidden();
|
||||
expect((await page.request.get('/api/logs')).status()).toBe(403);
|
||||
|
||||
// Subscribing to a feed the admin already has costs no second fetch: same feed, same
|
||||
// files, but Sam's own read state.
|
||||
await page.locator('#addFeed').click();
|
||||
await page.locator('#nurl').fill('http://127.0.0.1:8792/show.xml');
|
||||
await page.locator('#nsave').click();
|
||||
await expect(page.locator('.feed', { hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
await page.locator('.feed', { hasText: 'Test Show' }).click();
|
||||
await expect(page.locator('.ep').first()).toBeVisible({ timeout: 20_000 });
|
||||
// The admin read these earlier in this file; for Sam they are all still unread.
|
||||
const rows = await page.locator('.ep').count();
|
||||
await page.locator('.tabs button', { hasText: 'Unread' }).click();
|
||||
await expect(page.locator('.ep')).toHaveCount(rows);
|
||||
|
||||
// Two people now share this feed, so the page says so and Delete is honest about it.
|
||||
await expect(page.locator('#content .sub').first()).toContainText('shared with 1 other person');
|
||||
|
||||
await ctx.close();
|
||||
});
|
||||
|
||||
test('deleting a shared file warns that it is everyone\'s copy', async ({ page }) => {
|
||||
// Admin and Sam both subscribe to Test Show by now, and the daemon downloaded a file.
|
||||
await page.getByText('Test Show').click();
|
||||
await page.locator('.tabs button', { hasText: 'Downloaded' }).click();
|
||||
const row = page.locator('.ep').first();
|
||||
await expect(row).toBeVisible({ timeout: 20_000 });
|
||||
await row.click();
|
||||
|
||||
// An icon now; what it does, and to whom, is in its tooltip.
|
||||
const del = page.locator('#files button[data-a="del"]');
|
||||
await expect(del).toHaveAttribute('title', /^Delete for everyone \(shared with 1 other person/);
|
||||
|
||||
// Two prompts: the page's own, then the server's, because someone else has not played
|
||||
// it. Accept the first, decline the second, and the file must survive.
|
||||
const seen = [];
|
||||
page.on('dialog', d => {
|
||||
seen.push(d.message());
|
||||
if (seen.length === 1) d.accept();
|
||||
else d.dismiss();
|
||||
});
|
||||
await del.click();
|
||||
await expect.poll(() => seen.length, { timeout: 10_000 }).toBe(2);
|
||||
expect(seen[0]).toContain('shared with 1 other person');
|
||||
expect(seen[1]).toContain('one copy of this file');
|
||||
|
||||
await page.reload();
|
||||
await page.getByText('Test Show').click();
|
||||
await page.locator('.tabs button', { hasText: 'Downloaded' }).click();
|
||||
await expect(page.locator('.ep').first()).toBeVisible({ timeout: 20_000 });
|
||||
});
|
||||
|
||||
// Every row says "Admin" on its checkbox, so match the name exactly.
|
||||
const userRow = (page, name) =>
|
||||
page.locator('#modalCard [data-id]').filter({ has: page.locator('b', { hasText: new RegExp(`^${name}$`) }) });
|
||||
|
||||
async function openUsers(page) {
|
||||
await page.locator('#prefs').click();
|
||||
await page.locator('#gusers').click();
|
||||
await expect(userRow(page, 'admin')).toBeVisible();
|
||||
}
|
||||
|
||||
test('an admin adds someone, makes them an admin, and removes them', async ({ page }) => {
|
||||
await openUsers(page);
|
||||
await page.locator('#uname').fill('pat');
|
||||
await page.locator('#upass').fill('patpassword');
|
||||
await page.locator('#uadd').click();
|
||||
const row = userRow(page, 'pat');
|
||||
await expect(row).toBeVisible();
|
||||
// When each account was added and last signed in; the admin signed in with the token link.
|
||||
await expect(row).toContainText(/Added .* never signed in/);
|
||||
await expect(userRow(page, 'admin')).toContainText(/signed in \d+m ago/);
|
||||
await expect(row.locator('[data-a="admin"]')).not.toBeChecked();
|
||||
|
||||
await row.locator('[data-a="admin"]').check();
|
||||
// Not just the box: it has to have reached the database.
|
||||
await expect.poll(async () =>
|
||||
(await (await page.request.get('/api/users')).json()).find(u => u.name === 'pat')?.admin
|
||||
).toBe(true);
|
||||
|
||||
page.once('dialog', d => d.accept());
|
||||
await row.locator('[data-a="rm"]').click();
|
||||
await expect(row).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('the only admin cannot be demoted or removed', async ({ page }) => {
|
||||
await openUsers(page);
|
||||
await userRow(page, 'admin').locator('[data-a="admin"]').click();
|
||||
await expect(page.locator('.toast.bad')).toContainText('only admin');
|
||||
// Redrawn from the server, so the box is back.
|
||||
await expect(userRow(page, 'admin').locator('[data-a="admin"]')).toBeChecked();
|
||||
|
||||
const me = (await (await page.request.get('/api/users')).json()).find(u => u.name === 'admin');
|
||||
expect((await page.request.delete(`/api/users/${me.id}`)).status()).toBe(400);
|
||||
});
|
||||
|
||||
test('Settings exports your OPML and imports a pasted one', async ({ page }) => {
|
||||
await page.locator('#prefs').click();
|
||||
const [dl] = await Promise.all([
|
||||
page.waitForEvent('download'),
|
||||
page.locator('#modalCard a[title="Export OPML"]').click(),
|
||||
]);
|
||||
expect(dl.suggestedFilename()).toBe('ipx-subscriptions.opml');
|
||||
const out = require('fs').readFileSync(await dl.path(), 'utf8');
|
||||
for (const f of ['show.xml', 'pics.xml', 'multi.xml', 'subs.opml']) expect(out).toContain(f);
|
||||
// A feed from an OPML subscription comes back with the OPML itself, not on its own.
|
||||
expect(out).not.toContain('other.xml');
|
||||
|
||||
// One feed new to everyone, one the admin already has.
|
||||
await page.locator('#gopml').click();
|
||||
await page.locator('#opmlText').fill('<opml version="2.0"><head><title>t</title></head><body>' +
|
||||
'<outline text="Imported Show" xmlUrl="http://127.0.0.1:8792/imported.xml"/>' +
|
||||
'<outline text="Test Show" xmlUrl="http://127.0.0.1:8792/show.xml"/></body></opml>');
|
||||
await page.locator('#oimp').click();
|
||||
await expect(page.locator('.toast', { hasText: 'Subscribed to' }))
|
||||
.toHaveText('Subscribed to 1 feed(s), 1 you already had');
|
||||
// Named from the OPML's id until the first scan reads the feed's own title.
|
||||
await expect(page.locator('#feedlist .feed', { hasText: /Imported Show|imported-show/ }))
|
||||
.toBeVisible({ timeout: 20_000 });
|
||||
});
|
||||
|
||||
test('an uploaded OPML file imports, and a file that is not OPML is refused', async ({ page }) => {
|
||||
await page.locator('#prefs').click();
|
||||
await page.locator('#gopml').click();
|
||||
const pick = page.locator('#opmlFile');
|
||||
|
||||
// An RSS feed is XML but not OPML: refused in the page, and the picker lets go of it.
|
||||
await pick.setInputFiles({
|
||||
name: 'feed.xml', mimeType: 'application/xml',
|
||||
buffer: require('fs').readFileSync(require('path').join(__dirname, 'fixtures', 'show.xml')),
|
||||
});
|
||||
await page.locator('#oimp').click();
|
||||
await expect(page.locator('.toast.bad', { hasText: 'feed.xml is not an OPML file' })).toBeVisible();
|
||||
expect(await pick.evaluate(i => i.files.length)).toBe(0);
|
||||
|
||||
// Something that gets past the page's quick look is still refused by the server, untouched.
|
||||
const sneaky = await page.request.post('/api/opml', {
|
||||
data: { xml: '<rss version="2.0"><channel><title><opml></title></channel></rss>' },
|
||||
});
|
||||
expect(sneaky.status()).toBe(400);
|
||||
expect(await sneaky.text()).toContain('not an OPML file');
|
||||
|
||||
// A real one. Multi Show is already the admin's, so it counts as already had.
|
||||
await pick.setInputFiles({
|
||||
name: 'subs.opml', mimeType: 'text/x-opml',
|
||||
buffer: Buffer.from('<opml version="2.0"><head><title>t</title></head><body>' +
|
||||
'<outline text="Multi Show" xmlUrl="http://127.0.0.1:8792/multi.xml"/></body></opml>'),
|
||||
});
|
||||
await page.locator('#oimp').click();
|
||||
await expect(page.locator('.toast', { hasText: 'Subscribed to' }))
|
||||
.toHaveText('Subscribed to 0 feed(s), 1 you already had');
|
||||
await expect(page.locator('#modal.on')).toBeHidden();
|
||||
});
|
||||
|
||||
test('an export from one account imports into another', async ({ page, browser }) => {
|
||||
// Regression: import only added URLs the catalogue lacked and subscribed nobody, so importing
|
||||
// the admin's export into a second account did nothing at all.
|
||||
const { execFileSync } = require('child_process');
|
||||
const setup = require('./global-setup');
|
||||
const env = {
|
||||
...process.env,
|
||||
IPX_CONFIG: `${setup.root}/config/config.toml`,
|
||||
IPX_DATA_DIR: `${setup.root}/data`,
|
||||
};
|
||||
try {
|
||||
execFileSync('./target/debug/ipx', ['user', 'add', 'opal'], { input: 'opalpassword', env });
|
||||
} catch (e) {
|
||||
if (!String(e.stderr || e.stdout).includes('already exists')) throw e;
|
||||
}
|
||||
const ctx = await browser.newContext();
|
||||
const opal = await ctx.newPage();
|
||||
await opal.goto('/login');
|
||||
await opal.locator('#name').fill('opal');
|
||||
await opal.locator('#pw').fill('opalpassword');
|
||||
await opal.locator('button[type=submit]').click();
|
||||
await expect(opal.locator('#feedlist')).toContainText('No feeds.');
|
||||
|
||||
// Export used to hand anyone the whole catalogue. Opal has nothing yet, so gets nothing.
|
||||
const empty = await opal.request.get('/api/opml');
|
||||
expect(empty.status()).toBe(200);
|
||||
expect(await empty.text()).not.toContain('xmlUrl');
|
||||
|
||||
const urlsIn = xml => [...xml.matchAll(/xmlUrl="([^"]+)"/g)].map(m => m[1]).sort();
|
||||
const exported = await (await page.request.get('/api/opml')).text(); // the admin's
|
||||
const urls = urlsIn(exported);
|
||||
expect(urls.length).toBeGreaterThan(2);
|
||||
expect(await (await opal.request.post('/api/opml', { data: { xml: exported } })).json())
|
||||
.toEqual({ added: urls.length, already: 0 });
|
||||
expect(await (await opal.request.post('/api/opml', { data: { xml: exported } })).json())
|
||||
.toEqual({ added: 0, already: urls.length });
|
||||
|
||||
await opal.reload();
|
||||
await expect(opal.locator('.feed', { hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||
// The round trip closes: opal's own export now lists what the admin's did.
|
||||
expect(urlsIn(await (await opal.request.get('/api/opml')).text())).toEqual(urls);
|
||||
await ctx.close();
|
||||
});
|
||||
|
||||
test('ipx import subscribes the admin, and ipx export writes the feeds out', async () => {
|
||||
// Its own config and database. The CLI works in-process, and the suite's running daemon
|
||||
// reads config.toml once at start, so it would not see what the CLI added anyway.
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { execFileSync } = require('child_process');
|
||||
const setup = require('./global-setup');
|
||||
const dir = path.join(setup.root, 'cli');
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
fs.mkdirSync(path.join(dir, 'data'), { recursive: true });
|
||||
fs.writeFileSync(path.join(dir, 'config.toml'),
|
||||
`[general]\ndownload_dir = "${dir}/downloads"\nsocket = "${dir}/ipx.sock"\n`);
|
||||
const env = { ...process.env, IPX_CONFIG: path.join(dir, 'config.toml'), IPX_DATA_DIR: path.join(dir, 'data') };
|
||||
const ipx = (args, input) => execFileSync('./target/debug/ipx', args, { env, input, encoding: 'utf8' });
|
||||
|
||||
ipx(['user', 'add', 'boss'], 'bosspassword'); // the first account is the admin
|
||||
const opml = path.join(dir, 'in.opml');
|
||||
fs.writeFileSync(opml, '<opml version="2.0"><head><title>t</title></head><body>' +
|
||||
'<outline text="One" xmlUrl="http://127.0.0.1:8792/one.xml"/>' +
|
||||
'<outline text="Two" xmlUrl="http://127.0.0.1:8792/two.xml"/></body></opml>');
|
||||
expect(ipx(['import', opml])).toContain('subscribed boss to 2 feed(s); 0 already there');
|
||||
expect(ipx(['import', opml])).toContain('subscribed boss to 0 feed(s); 2 already there');
|
||||
|
||||
const out = path.join(dir, 'out.opml');
|
||||
ipx(['export', out]);
|
||||
const xml = fs.readFileSync(out, 'utf8');
|
||||
expect(xml).toContain('http://127.0.0.1:8792/one.xml');
|
||||
expect(xml).toContain('http://127.0.0.1:8792/two.xml');
|
||||
});
|
||||
|
||||
test('Popular lists what everyone here reads, but never a private feed', async ({ browser }) => {
|
||||
const { execFileSync } = require('child_process');
|
||||
const setup = require('./global-setup');
|
||||
const env = {
|
||||
...process.env,
|
||||
IPX_CONFIG: `${setup.root}/config/config.toml`,
|
||||
IPX_DATA_DIR: `${setup.root}/data`,
|
||||
};
|
||||
try {
|
||||
execFileSync('./target/debug/ipx', ['user', 'add', 'piper'], { input: 'piperpassword', env });
|
||||
} catch (e) {
|
||||
if (!String(e.stderr || e.stdout).includes('already exists')) throw e;
|
||||
}
|
||||
const ctx = await browser.newContext();
|
||||
const piper = await ctx.newPage();
|
||||
await piper.goto('/login');
|
||||
await piper.locator('#name').fill('piper');
|
||||
await piper.locator('#pw').fill('piperpassword');
|
||||
await piper.locator('button[type=submit]').click();
|
||||
await expect(piper.locator('#feedlist')).toContainText('No feeds.');
|
||||
|
||||
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
|
||||
const offered = piper.locator('#popular .childrow');
|
||||
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||
// An OPML is listed as the feeds inside it, and a key in a URL marks someone's paid feed.
|
||||
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
|
||||
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
|
||||
|
||||
// No URL reaches the page at all, so neither can a key, and the server holds the same line.
|
||||
const listed = await (await piper.request.get('/api/popular')).text();
|
||||
expect(listed).not.toContain('secret123');
|
||||
expect(listed).not.toContain('.xml');
|
||||
expect((await piper.request.post('/api/popular/paid-show')).status()).toBe(400);
|
||||
|
||||
// Popular is the top ten of the directory, and the directory is every listed feed A to Z,
|
||||
// with an OPML's feeds in place of the OPML in both.
|
||||
const dir = await (await piper.request.get('/api/directory')).json();
|
||||
const top = await (await piper.request.get('/api/popular')).json();
|
||||
const names = dir.map(p => (p.title || p.id).toLowerCase());
|
||||
expect(names).toEqual([...names].sort());
|
||||
const ids = dir.map(p => p.id);
|
||||
expect(ids).not.toContain('test-subscriptions');
|
||||
expect(ids).toEqual(expect.arrayContaining(['grouped-show', 'aardvark-radio']));
|
||||
expect(top.length).toBe(Math.min(10, dir.length));
|
||||
expect(top.every(t => ids.includes(t.id))).toBe(true);
|
||||
expect(ids).not.toContain('paid-show');
|
||||
|
||||
// Subscribe from the directory this time; the popular list shares the same rows.
|
||||
await piper.locator('#feedlist .place', { hasText: 'Directory' }).click();
|
||||
await expect(piper.locator('#count')).toContainText(`Directory: ${dir.length} feed`);
|
||||
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible();
|
||||
await expect(offered.filter({ hasText: /Grouped Show|grouped-show/ })).toBeVisible();
|
||||
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
|
||||
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
|
||||
|
||||
const row = async () =>
|
||||
(await (await piper.request.get('/api/popular')).json()).find(p => p.id === 'test-show');
|
||||
const before = await row();
|
||||
expect(before.subscribed).toBe(false);
|
||||
await offered.filter({ hasText: 'Test Show' }).locator('button[title="Subscribe"]').click();
|
||||
await expect(piper.locator('#feedlist .feed', { hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
// Everyone counts, you included: it stays listed, marked as yours, with one more subscriber.
|
||||
expect(await row()).toMatchObject({ subscribed: true, subscribers: before.subscribers + 1 });
|
||||
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
|
||||
await expect(offered.filter({ hasText: 'Test Show' }).locator('[title^="Subscribed"]')).toBeVisible();
|
||||
await expect(offered.filter({ hasText: 'Test Show' }).locator('button[title="Subscribe"]')).toHaveCount(0);
|
||||
|
||||
// All Subscriptions is every item from piper's feeds and only those: the admin's Picture
|
||||
// Blog is not among them.
|
||||
await piper.locator('#feedlist .place', { hasText: 'All Subscriptions' }).click();
|
||||
const first = piper.locator('.ep', { hasText: 'First Episode' });
|
||||
await expect(first).toBeVisible({ timeout: 20_000 });
|
||||
await expect(first.locator('.fd')).toHaveText('Test Show');
|
||||
await expect(piper.locator('.ep', { hasText: 'An Article' })).toHaveCount(0);
|
||||
await expect(piper.locator('#count')).toContainText('All Subscriptions:');
|
||||
await ctx.close();
|
||||
});
|
||||
|
||||
test('adding a feed scans it straight away', async ({ page }) => {
|
||||
await page.locator('#addFeed').click();
|
||||
await page.locator('#nurl').fill('http://127.0.0.1:8792/fresh.xml');
|
||||
await page.locator('#nsave').click();
|
||||
// Nobody pressed Scan. The scheduler's tick is a minute, so this is the add scanning it.
|
||||
await expect(page.locator('.ep', { hasText: 'Fresh Ep' })).toBeVisible({ timeout: 10_000 });
|
||||
});
|
||||
|
||||
test('a deleted file looks as if it was never downloaded', async ({ page }) => {
|
||||
// Other people subscribe to Picture Blog by now, so both prompts come; take them.
|
||||
page.on('dialog', d => d.accept());
|
||||
await page.locator('.feed', { hasText: 'Picture Blog' }).click();
|
||||
const row = page.locator('.ep', { hasText: 'An Article' });
|
||||
await expect(row).toBeVisible({ timeout: 20_000 });
|
||||
await row.click();
|
||||
await page.locator('#files button[data-a="del"]').click();
|
||||
|
||||
// No "reaped", no chip at all: just the way to get it again.
|
||||
await expect(row.locator('.kind.here')).toHaveCount(0);
|
||||
await expect(row).not.toContainText(/reaped/i);
|
||||
await row.click();
|
||||
await expect(page.locator('#files')).not.toContainText(/reaped/i);
|
||||
await expect(page.locator('#files button[title="Download to the server"]')).toBeVisible();
|
||||
});
|
||||
|
||||
test('one action, one icon: the toolbar, the page and every dialog agree', async ({ page }) => {
|
||||
const icon = loc => loc.locator('svg path').first().getAttribute('d');
|
||||
await page.locator('#feedlist .feed', { hasText: 'Test Show' }).first().click();
|
||||
|
||||
// Unsubscribe is a minus in the toolbar and the feed header, never the x that closes things.
|
||||
expect(await icon(page.locator('#content .acts [data-a="rm"]'))).toBe(await icon(page.locator('#tbRemove')));
|
||||
|
||||
// The toolbar's read and keep show the selected item's state, as its own buttons do, and follow
|
||||
// a change made from the toolbar.
|
||||
await page.locator('.ep').first().click();
|
||||
const pair = async a => [await icon(page.locator(a === 'read' ? '#tbRead' : '#tbFlag')),
|
||||
await icon(page.locator(`#detail [data-a="${a}"]`))];
|
||||
for (const a of ['read', 'flag']) { const [tb, own] = await pair(a); expect(tb).toBe(own); }
|
||||
const [kept] = await pair('flag');
|
||||
await page.locator('#tbFlag').click();
|
||||
await expect.poll(async () => { const [tb, own] = await pair('flag'); return tb === own && tb !== kept; }).toBe(true);
|
||||
await page.locator('#tbFlag').click(); // leave it as it was
|
||||
await expect.poll(async () => (await pair('flag'))[0]).toBe(kept);
|
||||
|
||||
// Every button in every dialog is an icon with its words in the tooltip.
|
||||
const dialogs = [
|
||||
() => page.locator('#addFeed').click(),
|
||||
() => page.locator('#prefs').click(),
|
||||
async () => { await page.locator('#prefs').click(); await page.locator('#gusers').click(); },
|
||||
async () => { await page.locator('#prefs').click(); await page.locator('#gopml').click(); },
|
||||
() => page.locator('#logs').click(),
|
||||
() => page.locator('#content .acts [data-a="settings"]').click(),
|
||||
() => page.locator('#content .acts [data-a="dl"]').click(),
|
||||
() => page.locator('#content .acts [data-a="rm"]').click(),
|
||||
];
|
||||
for (const open of dialogs) {
|
||||
await open();
|
||||
const btns = page.locator('#modalCard .btn');
|
||||
await expect(btns.first()).toBeVisible();
|
||||
for (const b of await btns.all()) {
|
||||
await expect(b.locator('svg')).toHaveCount(1);
|
||||
await expect(b).toHaveAttribute('title', /\S/);
|
||||
}
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(page.locator('#modal.on')).toBeHidden();
|
||||
}
|
||||
});
|
||||
|
||||
test('All Subscriptions marks everything read, across every feed', async ({ page }) => {
|
||||
const all = page.locator('#feedlist .place', { hasText: 'All Subscriptions' });
|
||||
await all.click();
|
||||
// Earlier tests read things; make sure something is unread. Opening an item reads it, and
|
||||
// its own button makes it unread again.
|
||||
await page.locator('.ep').first().click();
|
||||
await page.locator('#detail [data-a="read"][title="Mark unread"]').click();
|
||||
await expect(all.locator('.badge')).not.toHaveText('0');
|
||||
|
||||
page.once('dialog', d => d.accept());
|
||||
await page.locator('#content .acts [data-a="readall"]').click();
|
||||
await expect(all.locator('.badge')).toHaveText('0');
|
||||
await page.locator('.tabs button', { hasText: 'Unread' }).click();
|
||||
await expect(page.locator('.ep')).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('the item table sorts by any column, both ways, and remembers', async ({ page }) => {
|
||||
const all = page.locator('#feedlist .place', { hasText: 'All Subscriptions' });
|
||||
await all.click();
|
||||
const head = k => page.locator(`#list .ephead [data-sort="${k}"]`);
|
||||
const titles = () => page.locator('#eps .ep .t').allTextContents();
|
||||
// Byte order on lower case, which is what SQLite gives for lower(...).
|
||||
const cmp = (a, b) => (a.toLowerCase() < b.toLowerCase() ? -1 : a.toLowerCase() > b.toLowerCase() ? 1 : 0);
|
||||
const sorted = (t, dir) => JSON.stringify(t) === JSON.stringify([...t].sort((a, b) => cmp(a, b) * dir));
|
||||
await expect(page.locator('#eps .ep').nth(2)).toBeVisible({ timeout: 20_000 });
|
||||
expect(new Set(await titles()).size).toBeGreaterThan(2); // or both orders would prove nothing
|
||||
|
||||
await expect(head('title')).toHaveText('Title');
|
||||
await head('title').click();
|
||||
await expect.poll(async () => sorted(await titles(), 1)).toBe(true);
|
||||
await head('title').click();
|
||||
await expect.poll(async () => sorted(await titles(), -1)).toBe(true);
|
||||
|
||||
// Kept across a reload.
|
||||
await page.reload();
|
||||
await all.click();
|
||||
await expect(head('title').locator('.arr.desc')).toBeVisible();
|
||||
await expect.poll(async () => sorted(await titles(), -1)).toBe(true);
|
||||
|
||||
// Size has its own column; the file column is just what the file is.
|
||||
await expect(page.locator('#eps .ep .size', { hasText: /\d/ }).first()).toBeVisible();
|
||||
await expect(page.locator('#eps .ep .file', { hasText: /\d/ })).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('play in the Files pane plays once, in the player bar', async ({ page }) => {
|
||||
// Regression: the pane had an <audio> of its own, and playing it started the player bar too,
|
||||
// so the same file played twice at once.
|
||||
await page.locator('#feedlist .feed', { hasText: 'Test Show' }).first().click();
|
||||
await page.locator('.ep', { has: page.locator('.kind.here') }).first().click();
|
||||
await page.locator('#files [data-a="play"]').click();
|
||||
await expect(page.locator('#player')).toBeVisible();
|
||||
// The player bar's element doubles as a <video> so a video file has somewhere to show its
|
||||
// picture (see #audio's own comment), but there is still exactly one of it, and nothing else.
|
||||
await expect(page.locator('#audio')).toHaveCount(1);
|
||||
await page.locator('#pclose').click();
|
||||
});
|
||||
|
||||
test('someone the proxy signs in never sees the password page, and signs out through the proxy', async ({ page, browser }) => {
|
||||
// Signed in with the token, not by the proxy: Sign out stays ipx's own.
|
||||
expect((await (await page.request.get('/api/me')).json()).sign_out).toBeNull();
|
||||
|
||||
const ctx = await browser.newContext({ extraHTTPHeaders: { 'X-Test-User': 'proxied@example.com' } });
|
||||
const proxied = await ctx.newPage();
|
||||
// Regression: after Sign out, the password form showed to someone the proxy still vouched for.
|
||||
await proxied.goto('/login');
|
||||
await expect(proxied).toHaveURL(/:8791\/$/);
|
||||
await expect(proxied.locator('#who')).toContainText('proxied@example.com');
|
||||
expect(await (await proxied.request.get('/api/me')).json())
|
||||
.toMatchObject({ name: 'proxied@example.com', sign_out: '/signed-out-by-the-proxy' });
|
||||
await proxied.locator('#signout').click();
|
||||
await expect(proxied).toHaveURL(/\/signed-out-by-the-proxy$/);
|
||||
await ctx.close();
|
||||
});
|
||||
5
tests/ui/fixtures/aardvark.xml
Normal file
5
tests/ui/fixtures/aardvark.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Aardvark Radio</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Inside the OPML, and first in it and alphabetically.</description>
|
||||
<item><title>Aardvark Ep</title><guid>aa-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
BIN
tests/ui/fixtures/art.jpg
Normal file
BIN
tests/ui/fixtures/art.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.0 KiB |
BIN
tests/ui/fixtures/art2.jpg
Normal file
BIN
tests/ui/fixtures/art2.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 2.0 KiB |
BIN
tests/ui/fixtures/ep1.mp3
Normal file
BIN
tests/ui/fixtures/ep1.mp3
Normal file
Binary file not shown.
BIN
tests/ui/fixtures/ep2.mp3
Normal file
BIN
tests/ui/fixtures/ep2.mp3
Normal file
Binary file not shown.
5
tests/ui/fixtures/fresh.xml
Normal file
5
tests/ui/fixtures/fresh.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Fresh Show</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Added in the browser suite, and scanned by adding it.</description>
|
||||
<item><title>Fresh Ep</title><guid>fresh-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
5
tests/ui/fixtures/imported.xml
Normal file
5
tests/ui/fixtures/imported.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Imported Show</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Only ever arrives through an OPML import.</description>
|
||||
<item><title>Imported Ep</title><guid>imp-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
9
tests/ui/fixtures/multi.xml
Normal file
9
tests/ui/fixtures/multi.xml
Normal file
@@ -0,0 +1,9 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Multi Show</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>An item with more than one file.</description>
|
||||
<item><title>Two Files</title><guid>mu-1</guid>
|
||||
<pubDate>Mon, 01 Sep 2026 10:00:00 +0000</pubDate>
|
||||
<description>Audio and a picture.</description>
|
||||
<enclosure url="http://127.0.0.1:8792/ep2.mp3" length="40000" type="audio/mpeg"/>
|
||||
<enclosure url="http://127.0.0.1:8792/art2.jpg" length="3020" type="image/jpeg"/>
|
||||
</item></channel></rss>
|
||||
5
tests/ui/fixtures/other.xml
Normal file
5
tests/ui/fixtures/other.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Grouped Show</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Inside the OPML.</description>
|
||||
<item><title>Grouped Ep</title><guid>g-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
5
tests/ui/fixtures/paid.xml
Normal file
5
tests/ui/fixtures/paid.xml
Normal file
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Paid Show</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>Subscribed with a key in its URL, so it must never be offered to anyone else.</description>
|
||||
<item><title>Paid Ep</title><guid>paid-1</guid><description>x</description></item>
|
||||
</channel></rss>
|
||||
8
tests/ui/fixtures/pics.xml
Normal file
8
tests/ui/fixtures/pics.xml
Normal file
@@ -0,0 +1,8 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0"><channel><title>Picture Blog</title><link>http://127.0.0.1:8792/</link>
|
||||
<description>A text blog whose entries carry a header image, as Substack does.</description>
|
||||
<item><title>An Article</title><guid>pic-1</guid>
|
||||
<pubDate>Mon, 01 Sep 2026 10:00:00 +0000</pubDate>
|
||||
<description><p>Words, not audio.</p></description>
|
||||
<enclosure url="http://127.0.0.1:8792/art.jpg" length="3020" type="image/jpeg"/></item>
|
||||
</channel></rss>
|
||||
19
tests/ui/fixtures/serve.js
Normal file
19
tests/ui/fixtures/serve.js
Normal file
@@ -0,0 +1,19 @@
|
||||
// Serves the fixture feeds so the daemon under test has something real to scan.
|
||||
const http = require('http');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const dir = __dirname;
|
||||
const port = Number(process.env.FIXTURE_PORT || 8792);
|
||||
|
||||
http.createServer((req, res) => {
|
||||
const name = decodeURIComponent(req.url.split('?')[0].replace(/^\//, '')) || 'index';
|
||||
const file = path.join(dir, path.basename(name));
|
||||
fs.readFile(file, (err, body) => {
|
||||
if (err) { res.writeHead(404).end('no'); return; }
|
||||
const type = file.endsWith('.mp3') ? 'audio/mpeg'
|
||||
: file.endsWith('.opml') ? 'text/x-opml' : 'application/xml';
|
||||
res.writeHead(200, { 'content-type': type, 'content-length': body.length });
|
||||
res.end(body);
|
||||
});
|
||||
}).listen(port, '127.0.0.1', () => console.log(`fixtures on ${port}`));
|
||||
15
tests/ui/fixtures/show.xml
Normal file
15
tests/ui/fixtures/show.xml
Normal file
@@ -0,0 +1,15 @@
|
||||
<?xml version="1.0"?>
|
||||
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd">
|
||||
<channel><title>Test Show</title><link>http://127.0.0.1:8792/</link><description>A fixture feed.</description>
|
||||
<itunes:image href="http://127.0.0.1:8792/art.png"/>
|
||||
<item><title>First Episode</title><guid>ui-1</guid>
|
||||
<pubDate>Mon, 01 Sep 2026 10:00:00 +0000</pubDate>
|
||||
<description><p>Show notes for the first one.</p></description>
|
||||
<itunes:duration>1830</itunes:duration><itunes:season>1</itunes:season><itunes:episode>1</itunes:episode>
|
||||
<enclosure url="http://127.0.0.1:8792/ep1.mp3" length="40000" type="audio/mpeg"/></item>
|
||||
<item><title>Second Episode</title><guid>ui-2</guid>
|
||||
<pubDate>Mon, 08 Sep 2026 10:00:00 +0000</pubDate>
|
||||
<description>Notes for the second.</description>
|
||||
<itunes:duration>900</itunes:duration>
|
||||
<enclosure url="http://127.0.0.1:8792/ep1.mp3?2" length="40000" type="audio/mpeg"/></item>
|
||||
</channel></rss>
|
||||
5
tests/ui/fixtures/subs.opml
Normal file
5
tests/ui/fixtures/subs.opml
Normal file
@@ -0,0 +1,5 @@
|
||||
<opml version="2.0"><head><title>Test Subscriptions</title></head>
|
||||
<body><outline text="Folder">
|
||||
<outline type="rss" text="Aardvark Radio" xmlUrl="http://127.0.0.1:8792/aardvark.xml"/>
|
||||
<outline type="rss" text="Grouped Show" xmlUrl="http://127.0.0.1:8792/other.xml"/>
|
||||
</outline></body></opml>
|
||||
70
tests/ui/global-setup.js
Normal file
70
tests/ui/global-setup.js
Normal file
@@ -0,0 +1,70 @@
|
||||
// Builds a scratch config and data dir so the browser tests drive a real daemon with
|
||||
// known feeds, rather than whatever happens to be on the machine.
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const os = require('os');
|
||||
|
||||
const root = path.join(os.tmpdir(), 'ipx-ui-test');
|
||||
const TOKEN = 'testtokentesttokentesttoken12345'; // fixed, so tests need not scrape a log
|
||||
|
||||
// Called from playwright.config.js at load time, NOT as globalSetup: Playwright starts
|
||||
// webServer *before* globalSetup, so a config written there does not exist yet when the
|
||||
// daemon launches -- it would fall back to the real config and fight the live daemon.
|
||||
// Playwright imports this config again in every worker process, so prepare() runs more
|
||||
// than once per suite. Wiping on the second call deleted the data directory out from under
|
||||
// the running daemon: it kept serving from the unlinked inode, while anything else opening
|
||||
// that path -- the CLI, a query -- got a brand new empty database and disagreed with it.
|
||||
function prepare() {
|
||||
// Only the process that launches the run may wipe. A worker gets TEST_WORKER_INDEX.
|
||||
if (process.env.TEST_WORKER_INDEX !== undefined || process.env.PW_WORKER_INDEX !== undefined) {
|
||||
return;
|
||||
}
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
for (const d of ['config', 'data', 'downloads']) {
|
||||
fs.mkdirSync(path.join(root, d), { recursive: true });
|
||||
}
|
||||
fs.writeFileSync(path.join(root, 'config', 'config.toml'), `
|
||||
[general]
|
||||
download_dir = "${path.join(root, 'downloads')}"
|
||||
socket = "${path.join(root, 'ipx.sock')}"
|
||||
schedule = "every 60m"
|
||||
max_new_per_check = 1
|
||||
|
||||
[torrent]
|
||||
enabled = false
|
||||
|
||||
[web]
|
||||
enabled = true
|
||||
bind = "127.0.0.1:8791"
|
||||
token = "${TOKEN}"
|
||||
# The proxy path, for tests that send the header themselves: the daemon sees them at 127.0.0.1.
|
||||
trusted_header = "X-Test-User"
|
||||
trusted_proxies = ["127.0.0.1"]
|
||||
sign_out_url = "/signed-out-by-the-proxy"
|
||||
|
||||
[feeds.test-show]
|
||||
url = "http://127.0.0.1:8792/show.xml"
|
||||
auto_download = true
|
||||
|
||||
# Downloads its image, so the UI has a file that is not playable to deal with.
|
||||
[feeds.picture-blog]
|
||||
url = "http://127.0.0.1:8792/pics.xml"
|
||||
auto_download = true
|
||||
media_types = ["image"]
|
||||
|
||||
[feeds.multi-show]
|
||||
url = "http://127.0.0.1:8792/multi.xml"
|
||||
auto_download = true
|
||||
|
||||
[feeds.test-subscriptions]
|
||||
url = "http://127.0.0.1:8792/subs.opml"
|
||||
auto_download = false
|
||||
|
||||
# A key in its URL, like a Patreon feed: someone's paid subscription, never offered to others.
|
||||
[feeds.paid-show]
|
||||
url = "http://127.0.0.1:8792/paid.xml?auth=secret123"
|
||||
auto_download = false
|
||||
`);
|
||||
}
|
||||
|
||||
module.exports = { prepare, root, TOKEN };
|
||||
2013
web/index.html
Normal file
2013
web/index.html
Normal file
File diff suppressed because it is too large
Load Diff
BIN
web/ipodderx-icon.jpg
Normal file
BIN
web/ipodderx-icon.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 9.3 KiB |
BIN
web/ipodderx-icon.png
Normal file
BIN
web/ipodderx-icon.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
93
web/login.html
Normal file
93
web/login.html
Normal file
@@ -0,0 +1,93 @@
|
||||
<title>Sign in — iPodderX</title>
|
||||
<link rel="icon" href="/icon.png">
|
||||
<style>
|
||||
:root {
|
||||
--bg:#0e131b; /* the screen's navy (#314B74), taken right down */
|
||||
--panel:#151c27;
|
||||
--panel2:#1c2431;
|
||||
--raise:#25303f;
|
||||
--line:#2c3849;
|
||||
--fg:#f5f5f5; /* #F5F5F5 device highlight */
|
||||
--dim:#95a0b1; /* #95A0B1 straight from the icon's blue-grey */
|
||||
--faint:#7a8799; /* lifted from the icon ramp until it clears AA at small sizes */
|
||||
--accent:#92b2e6; /* #92B2E6 the screen blue */
|
||||
--accent2:#f49e2c; /* #F49E2C the EQ bars */
|
||||
--ink:#0e131b; /* text on an accent fill */
|
||||
--good:#6fbf8b;
|
||||
--warn:#f49e2c; /* the amber doubles as the pending colour */
|
||||
--bad:#e2705f;
|
||||
--shadow:0 8px 28px rgba(6,10,16,.55);
|
||||
--r:10px;
|
||||
}
|
||||
:root[data-theme="light"] {
|
||||
--bg:#f2f4f7;
|
||||
--panel:#ffffff; /* #FFFFFF device body */
|
||||
--panel2:#e9edf3;
|
||||
--raise:#dde3ec;
|
||||
--line:#d6d6d6; /* #D6D6D6 device edge */
|
||||
--fg:#1a1a1a; /* #1A1A1A icon outline */
|
||||
--dim:#606060; /* #606060 */
|
||||
--faint:#767676; /* between the icon's #929292 and #606060, to clear AA */
|
||||
--accent:#2d5391; /* #2D5391 the deep screen blue reads better on white */
|
||||
--accent2:#b06f10;
|
||||
--ink:#ffffff;
|
||||
--good:#2f7d4f;
|
||||
--warn:#b06f10;
|
||||
--bad:#b3402f;
|
||||
--shadow:0 8px 28px rgba(45,83,145,.14);
|
||||
}
|
||||
*{box-sizing:border-box}
|
||||
html,body{height:100%}
|
||||
body{
|
||||
margin:0;display:grid;place-items:center;background:var(--bg);color:var(--fg);
|
||||
font:14.5px/1.55 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;padding:20px;
|
||||
}
|
||||
form{
|
||||
width:min(360px,100%);background:var(--panel);border:1px solid var(--line);
|
||||
border-radius:14px;padding:22px;box-shadow:var(--shadow);
|
||||
}
|
||||
/* The one place the 2004 icon is shown at the size it was drawn for. */
|
||||
.brand{display:flex;flex-direction:column;align-items:center;gap:6px;margin-bottom:20px}
|
||||
.brand img{width:96px;height:auto}
|
||||
h1{font-size:21px;margin:0;font-weight:650;letter-spacing:-.01em}
|
||||
label{display:block;font-size:12px;color:var(--dim);margin:0 0 4px}
|
||||
input{
|
||||
width:100%;background:var(--bg);border:1px solid var(--line);color:var(--fg);
|
||||
border-radius:8px;padding:9px 11px;font:inherit;margin-bottom:13px;
|
||||
}
|
||||
input:focus{outline:0;border-color:var(--accent)}
|
||||
button{
|
||||
width:100%;font:inherit;font-weight:600;cursor:pointer;color:var(--ink);
|
||||
background:var(--accent);border:0;border-radius:8px;padding:10px;
|
||||
}
|
||||
.msg{color:var(--bad);font-size:13px;min-height:19px;margin:10px 0 0;text-align:center}
|
||||
button:focus-visible{outline:2px solid var(--accent);outline-offset:2px}
|
||||
</style>
|
||||
|
||||
<form id="f">
|
||||
<div class="brand"><img src="/icon.png" alt=""><h1>iPodderX</h1></div>
|
||||
<label for="name">Name</label>
|
||||
<input id="name" name="name" autocomplete="username" autofocus required>
|
||||
<label for="pw">Password</label>
|
||||
<input id="pw" name="password" type="password" autocomplete="current-password" required>
|
||||
<button type="submit">Sign in</button>
|
||||
<p class="msg" id="msg"></p>
|
||||
</form>
|
||||
|
||||
<script>
|
||||
document.getElementById('f').onsubmit = async e => {
|
||||
e.preventDefault();
|
||||
const msg = document.getElementById('msg');
|
||||
msg.textContent = '';
|
||||
const r = await fetch('/api/login', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
name: document.getElementById('name').value,
|
||||
password: document.getElementById('pw').value,
|
||||
}),
|
||||
});
|
||||
if (r.ok) location.href = '/';
|
||||
else msg.textContent = await r.text() || 'Sign in failed';
|
||||
};
|
||||
</script>
|
||||
Reference in New Issue
Block a user