11 Commits

Author SHA1 Message Date
2af57065c6 Release 0.5.1
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 02:15:53 +00:00
564b011c7a Give the folder triangle a wider gutter
The feed list's left padding grows from 8 to 16 px and the triangle's
button from 16 to 24 px wide, so it is no longer cramped against the
folder's art. Everything in the list shifts together, so feeds still line
up with the places above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 02:13:32 +00:00
9eb7aadced Release 0.5.0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 02:06:23 +00:00
c6bceaef37 Docs: a slow migration and a CLI run at the same time
Every ipx command migrates when it opens the database, so the healthcheck
collided with the daemon while it dropped the old entries columns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 01:59:05 +00:00
dc63d6acaf Cut what the audit found: dead columns, one-time upgrades, three deps
Works through TODO.md from the 2026-09-12 over-engineering audit. Drops the
entries.read/flagged/position columns (migrate() removes them from older
databases), migrate_opml_children, the legacy interval_mins key, the
contrib/ systemd units, test-only Db wrappers, a duplicate token generator,
redundant logbuf visitors, unused page state and CSS, and the infer, dirs
and tokio-stream dependencies. The icon is served once as /icon.png instead
of inlined four times, taking about 94 KB off the two pages.

The adoption's subscription half was not dead: it gives a fresh install's
first admin the config's feeds. It stays as adopt_catalogue, now tested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 01:55:44 +00:00
8937f35f00 TODO.md: cleared, the design pass is done
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 01:31:08 +00:00
0990f2a90d Design pass on the web UI: amber for new, EQ bars, keyboard feed list
Works through TODO.md from the 2026-09-11 review. Unread badges, dots and
download bars take the icon's amber; the playing item is marked by EQ bars
that move only while it plays. The feed list is usable from the keyboard,
focus rings show everywhere, and folders get a mosaic of their shows' art
with the triangle hung in the margin. Sentence-case labels, fewer bold
weights, tinted initials tiles, shorter header lines, "Kept" everywhere,
and the list gets the room the empty panes had. Reduced motion is honoured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-11 22:24:41 +00:00
0cc002cdfa todo.md: folders in the sidebar; drop the system-theme item
The disclosure triangle, looked at closely: the feed list cannot be
reached from the keyboard (span and div, no tabindex), every feed is
pushed 28 px right for a slot only folders use, the target is 18 px,
shows barely nest under their folder, a folder looks like a feed, and
a selected feed's placeholder tile vanishes in Dark and Light.

Following the system light or dark setting is off the list, by
choice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wi22VSVrkAvqNj61eqHsm9
2026-09-11 19:39:49 +00:00
2c9e899762 todo.md: the design review's list, to do later
A review of the web UI against screenshots of every view in all three
themes. The palette and Classic carry the iPodderX identity; Dark and
Light do not. First three: amber for new (unread badges, dots,
download progress), EQ bars as the playing marker, and toolbar focus
rings that overflow:hidden currently clips. The rest is weights,
sentence-case labels, pane sizes, header lines, wording, system theme,
reduced motion, art, sign-in and the Settings export/import icons.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wi22VSVrkAvqNj61eqHsm9
2026-09-11 18:53:42 +00:00
2e416f96cf Patreon creators split into their shows; filters follow settings
A Patreon token pasted into Add feed, or a creator link without
&show=, becomes a folder of that creator's shows, found through
Patreon's web API and kept in step like a subscribed OPML (sync_group,
split out of sync_opml). A creator already read as one feed is split
too: each show takes over the files and read state it held
(Db::adopt). A creator with one show stays a plain feed.

Filter verdicts are judged again every scan, so turning on Allow
explicit brings skipped items back. Add feed has an explicit box.
Feeds in a group follow your settings on the group, as its dialog
said. A new feed no longer takes the id of a removed one at a
different URL and shows its old items. See CHANGELOG.md [Unreleased]
and docs/history.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wi22VSVrkAvqNj61eqHsm9
2026-09-11 18:24:47 +00:00
9269aa99f7 README: a short overview that points into docs/
It described the layout from before 0.4.0 (items across the top, a
player below), and carried long sections on OPML, the log view and
this server's own deploy steps, all of which docs/ and CLAUDE.md cover.
It now says what ipx does, how to run it with Docker or from source,
the first sign-in, the TLS caveat, where the docs are, and the tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0173mGu6rK18Ne7UGTwAaVJV
2026-09-11 17:36:34 +00:00
24 changed files with 1010 additions and 518 deletions

View File

@@ -10,6 +10,72 @@ The long form, with what was wrong before and how it was found, is in
## [Unreleased] ## [Unreleased]
## [0.5.1] - 2026-09-12
### Fixed
- The triangle that opens an OPML or Patreon folder was cramped against the folder's art. It has
more room now, and a wider target to click.
## [0.5.0] - 2026-09-12
### Added
- A Patreon token pasted into Add feed, or a creator's RSS link without `&show=`, becomes a folder
of that creator's shows, kept in step on every scan like a subscribed OPML. A creator with only
one show stays a plain feed. One already added as a single long feed is split into its shows on
its next scan, keeping its files and what you had read.
- Add feed has an "Allow items marked explicit" box, so a new feed's first scan no longer skips
every explicit item.
### Changed
- Unread counts, unread dots and download progress are amber, the colour of the icon's EQ bars.
Blue is kept for the primary action and links, so a count no longer looks like a button.
- What is playing is marked by small EQ bars, in its row and in the player. They move only while it
plays.
- A folder in the sidebar shows its first four shows' art as a mosaic, and its shows sit under its
title. Only folders have a triangle, so every feed lines up with Directory and Popular above.
- Feeds without art get initials in a colour of their own, instead of all the same grey.
- The Flagged tab is called Kept, as the Keep button and Settings already said.
- A feed's header is one short line; when it checks next is in its tooltip.
- The item list takes more of the window, and the Files pane shows only when the item has files.
- Column headings and tags are in sentence case, and fewer things are bold.
- Unsubscribe is a round button beside the feed's other actions.
- Export and Import in Settings say what they do.
- The sign-in page shows the original icon large.
- Nothing animates when your system asks for reduced motion.
- The pages are about 90 KB smaller: the icon is served once instead of written into each.
- A web token generated for a new install is 64 characters instead of 32.
- The README is a short overview of what ipx does and how to run it, and points into `docs/` for
the rest. It still described the layout from before 0.4.0.
### Removed
- The systemd units in `contrib/`. Run ipx with Docker, or point a unit of your own at
`ipx daemon`.
- Upgrading from before 0.3.0 directly: what was read, kept or part-played before accounts is no
longer carried over to the admin, and OPML feeds that old versions wrote into `config.toml` are
no longer moved out of it. Upgrade through 0.4.0 first.
- `interval_mins` in `config.toml` is ignored; use `schedule`.
### Fixed
- The feed list works from the keyboard: Tab reaches every feed, Enter opens it, and Right and Left
open and close a folder. Every button shows where the focus is, including in the toolbar, which
used to clip the ring.
- "1 items" reads "1 item".
- A selected feed without art no longer loses its initials tile in Dark and Light.
- The player shows the feed's initials when there is no art, not the episode's.
- Turning on Allow explicit, or changing keywords or auto-download, brings back what those settings
had skipped on the feed's next scan. Before, an item was judged once, when first seen, and a
skipped one stayed skipped whatever you changed.
- Feeds inside an OPML or a Patreon creator follow your settings on the folder unless you set their
own, as the folder's settings dialog said they did. Before, the folder's settings reached nothing
inside it.
- A new feed no longer takes the name of one you removed earlier and shows that feed's old items.
Re-adding the same feed still gets its old name, and its history, back.
## [0.4.0] - 2026-09-11 ## [0.4.0] - 2026-09-11
### Added ### Added
@@ -199,7 +265,9 @@ The long form, with what was wrong before and how it was found, is in
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout. - Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`. - `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...main [unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.1...main
[0.5.1]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.0...v0.5.1
[0.5.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...v0.5.0
[0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0 [0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0
[0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0 [0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0
[0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0 [0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0

View File

@@ -99,9 +99,9 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every * **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
subscriber. Two feeds publishing the same URL means only the first one scanned shows it. subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
* **`entries.read`, `entries.flagged` and `entries.position` are dead columns.** Read state lives in * **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
`entry_state` per user. Two bugs have already come from queries still reading the old ones and `position` columns from before accounts; two bugs came from queries still reading them
(retention, and the entry pruner) — grep before adding a third. (retention, and the entry pruner), and `migrate()` now drops them.
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once; * **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are `subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
derived and never written to config. derived and never written to config.
@@ -115,6 +115,10 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
fighting over the database, with the stale one still holding the port. fighting over the database, with the stale one still holding the port.
* `/api/settings` answering `200` does **not** mean the worker is alive — it is a different task. * `/api/settings` answering `200` does **not** mean the worker is alive — it is a different task.
Probe the control socket (`ipx status`) to check that. Probe the control socket (`ipx status`) to check that.
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait
for `daemon started` in the log. Copy `state.db` aside before deploying one.
## House style ## House style

34
Cargo.lock generated
View File

@@ -436,17 +436,6 @@ dependencies = [
"shlex", "shlex",
] ]
[[package]]
name = "cfb"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a347dcabdae9c31b0825fd6a8bed285ec9c2acb89c47827126d52fa4f59cece3"
dependencies = [
"fnv",
"uuid",
"web-time",
]
[[package]] [[package]]
name = "cfg-if" name = "cfg-if"
version = "1.0.4" version = "1.0.4"
@@ -867,15 +856,6 @@ dependencies = [
"dirs-sys", "dirs-sys",
] ]
[[package]]
name = "dirs"
version = "7.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8d57d423b3c82e89b9a24ca3091fee61f456a26edbd28d26c65906f4bc1dcd8f"
dependencies = [
"dirs-sys",
]
[[package]] [[package]]
name = "dirs-sys" name = "dirs-sys"
version = "0.5.0" version = "0.5.0"
@@ -1608,15 +1588,6 @@ dependencies = [
"serde_core", "serde_core",
] ]
[[package]]
name = "infer"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f4200d433cbd5178df7797c9c2e75b348b728e39631cf14520d1e2fc424201f4"
dependencies = [
"cfb",
]
[[package]] [[package]]
name = "intervaltree" name = "intervaltree"
version = "0.2.7" version = "0.2.7"
@@ -1634,7 +1605,7 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]] [[package]]
name = "ipx" name = "ipx"
version = "0.4.0" version = "0.5.1"
dependencies = [ dependencies = [
"ammonia", "ammonia",
"anyhow", "anyhow",
@@ -1643,9 +1614,7 @@ dependencies = [
"axum", "axum",
"chrono", "chrono",
"clap", "clap",
"dirs",
"futures-util", "futures-util",
"infer",
"librqbit", "librqbit",
"opml", "opml",
"percent-encoding", "percent-encoding",
@@ -1656,7 +1625,6 @@ dependencies = [
"serde", "serde",
"serde_json", "serde_json",
"tokio", "tokio",
"tokio-stream",
"toml", "toml",
"tower", "tower",
"tower-http 0.7.1", "tower-http 0.7.1",

View File

@@ -1,6 +1,6 @@
[package] [package]
name = "ipx" name = "ipx"
version = "0.4.0" version = "0.5.1"
edition = "2024" edition = "2024"
[dependencies] [dependencies]
@@ -11,9 +11,7 @@ atom_syndication = "0.12.10"
axum = "0.8.9" axum = "0.8.9"
chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] } chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] }
clap = { version = "4.6.6", features = ["derive"] } clap = { version = "4.6.6", features = ["derive"] }
dirs = "7.0.0"
futures-util = { version = "0.3.34", default-features = false, features = ["std"] } futures-util = { version = "0.3.34", default-features = false, features = ["std"] }
infer = "0.22.0"
librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] } librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] }
opml = "1.1.6" opml = "1.1.6"
percent-encoding = "2.3.2" percent-encoding = "2.3.2"
@@ -24,7 +22,6 @@ rusqlite = { version = "0.40.2", features = ["bundled"] }
serde = { version = "1.0.229", features = ["derive"] } serde = { version = "1.0.229", features = ["derive"] }
serde_json = "1.0.151" serde_json = "1.0.151"
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] } tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
tokio-stream = { version = "0.1.19", features = ["sync"] }
toml = "1.1.5" toml = "1.1.5"
tower = { version = "0.5.3", features = ["util"] } tower = { version = "0.5.3", features = ["util"] }
tower-http = { version = "0.7.1", features = ["fs"] } tower-http = { version = "0.7.1", features = ["fs"] }

144
README.md
View File

@@ -1,123 +1,83 @@
# ipodderx-rs # ipodderx-rs
A headless podcatcher: scans RSS/Atom feeds, downloads enclosures (HTTP and BitTorrent), files them A self-hosted podcatcher for a household. It checks your feeds, downloads the episodes, and serves
into per-feed folders, and reaps old files to stay under a disk quota. Runs as a one-shot CLI or as a web UI modelled on the 2004 Mac app **iPodderX**, for any number of people sharing one copy of
a daemon with a web UI, serving any number of people from one copy of the data. the files. One Rust binary, `ipx`, is both the daemon and the command line.
## Lineage It is a rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python engine
behind iPodderX (2004-2008, Ray Slakinski & August Trometer).
A modern Rust rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python 2 ## What it does
engine behind **iPodderX** (2004-2008, Ray Slakinski & August Trometer), open-sourced under the MIT
License in 2010.
What carries over: the feed scan and TTL handling, GUID/URL dedupe, per-feed and per-date download - **The web UI.** It has a toolbar, and a feed list that opens with Directory, Popular and All
folders, keyword filters, the explicit-content filter, torrent enclosures, and "SmartSpace" -- the Subscriptions. Items sit in a sortable table with a Files pane, and there is a player bar. It
oldest-first disk quota reaper. comes in Dark, Light and Classic themes, and works on a phone.
- **Several people, one copy.** Each person has their own subscriptions and their own read, kept
and playback state. There is one file on disk per episode, however many people want it. People
sign in with a password or through a proxy (Cloudflare Zero Trust or Authentik), and admins
manage accounts and settings.
- **Scanning.** Feeds are checked on a schedule, globally or per feed, and a feed's own TTL is
honoured. Keyword, explicit-content and media-type filters decide what is downloaded, with a cap
on new downloads per scan.
- **Downloads.** Files come over HTTP or BitTorrent and are filed into a folder per feed.
Retention deletes the oldest files to stay under a disk quota or an age limit, and never touches
an item someone has kept.
- **OPML.** You can import and export your own subscriptions. You can also subscribe to an OPML
URL, which keeps a whole list in step as a folder.
What does not: iTunes and iPhoto export via AppleScript, text-to-speech enclosures, the Windows ## Run it
WMP/COM paths, XML plists and Python pickles for state, the `directory.iPodderX.com` survey ping,
3DES-encrypted preferences, and the `printMSG` stdout protocol -- replaced by a JSON-lines socket.
## Quick start With Docker:
```sh
docker build -t ipodderx .
docker compose up -d
```
`docker-compose.yml` is set up for the author's own server. Point its `image` and its three volumes
(`/config`, `/data` and `/downloads`) at yours first. The UI is on port 8099. BitTorrent uses 6881
over TCP and UDP. Files are written as `PUID`/`PGID`, 99:100 by default.
From source:
```sh ```sh
cargo build --release cargo build --release
install -m755 target/release/ipx ~/.cargo/bin/ ./target/release/ipx daemon
ipx add https://atp.fm/rss # subscribe
ipx fetch # scan and download
ipx daemon # scheduler, control socket and web UI
``` ```
On first start with `[web] enabled = true` the daemon mints a token, writes it to config.toml and The first start creates **admin / ipodderx**. Sign in at `/login`, then change it:
prints the URL to open. A database with no accounts starts with **admin / ipodderx** at `/login` --
change it with `echo -n '<password>' | ipx user passwd admin`. ```sh
echo -n 'a good password' | ipx user passwd admin
```
The UI is plain HTTP, so put TLS in front of it if it is reachable from outside your network.
## Documentation ## Documentation
| | | | | |
|---|---| |---|---|
| [docs/configuration.md](docs/configuration.md) | Every config key, paths, environment variables | | [docs/configuration.md](docs/configuration.md) | Every config key, path and environment variable |
| [docs/cli.md](docs/cli.md) | Every command, including `ipx user` | | [docs/cli.md](docs/cli.md) | Every command, including `ipx user` |
| [docs/users.md](docs/users.md) | Accounts, and what several people share | | [docs/users.md](docs/users.md) | Accounts, and what several people share |
| [docs/sso.md](docs/sso.md) | Cloudflare Zero Trust or Authentik in front | | [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik |
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, socket, HTTP API | | [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API |
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release | | [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
| [docs/history.md](docs/history.md) | How it was built: the long form, with what was wrong and why | | [docs/history.md](docs/history.md) | How it was built, with what was wrong and why |
| [CLAUDE.md](CLAUDE.md) | Notes for anyone (or anything) working on the code | | [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed |
## The web UI
`ipx daemon` serves it in the same process, so it reads SQLite and the event bus directly.
Feeds down the side; the selected feed's items across the top; the selected item's text and its
enclosures below, which is where you play, download or delete them. The divider drags and its
position is remembered. Playback serves Range requests, so seeking works. An OPML subscription is a
collapsible folder whose page lists the feeds inside it.
An item may carry several enclosures; all of them appear below, and anything that is not audio or
video gets a View link rather than a player -- the publisher's copy until it is downloaded, the
local one after. Opening an item marks it read. Show notes are untrusted feed HTML, sanitized with
`ammonia` server-side before they reach the page.
The **Log** button shows the running daemon live in four tabs: *Daemon I/O* is the control protocol
itself, every command in and event out; *Scans* is feed and download activity; *HTTP* is web
requests; *All* is everything, with level and text filters and a copy button. It reads a ring buffer
held in the process, not a file, so it works the same under Docker.
It is plain HTTP. On a LAN bind the token and everything else cross the network in the clear, and a
feed URL can itself carry a credential. Put TLS in front of it if that matters.
## OPML
**Importing and exporting** a file copies subscriptions in or out once: `ipx import subs.opml`,
`ipx export subs.opml`, or Settings → Subscriptions in the UI.
**Subscribing to an OPML URL** is a live subscription, as iPodderX had. Add the OPML's URL like any
other feed; every scan re-reads it and keeps your list in step. The feeds inside are not written to
config.toml -- the OPML is the source of truth, so they are re-derived each scan and held in the
database. They show as a folder, download into one nested folder, and inherit the subscription's
settings until you change one, which gives it its own entry.
When a feed drops out of the OPML upstream, it is unsubscribed and removed -- unless it has
downloads, in which case it is kept and flagged in the UI as no longer listed. A downloaded file is
never left behind with nothing explaining where it came from.
## Docker
```sh
docker buildx build --tag 192.168.1.130:5000/ipodderx:latest . --push
docker compose pull ipodderx && docker compose up -d ipodderx
docker compose logs -f ipodderx # the first start prints the default admin password
```
`docker-compose.yml` runs the image from the registry above rather than building it, so build and
push first; change the tag in both places to use another registry. It mounts `/config` (config.toml),
`/data` (state.db) and `/downloads` from this install's host paths, which you will want to change for
yours. It publishes 8099 for the
UI and 6881 (TCP **and** UDP -- DHT needs the UDP side), and sets `PUID`/`PGID` to `99:100` so files
land owned the way Unraid shares expect. The healthcheck runs `ipx status` through the control
socket, so it catches a daemon that is alive but wedged rather than merely one that has died.
## Running it as a service
`contrib/` has a systemd user unit for the daemon, and a timer plus one-shot service if you would
rather run periodic scans with no daemon -- in which case there is no socket for a UI to attach to.
## Tests ## Tests
```sh ```sh
cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state
node tests/page-smoke.js # the page script loads without throwing node tests/page-smoke.js # the page script loads without throwing
npx playwright test # a real browser against a real daemon npx playwright test # a real browser against a real daemon on fixture feeds
``` ```
`npm install` gets the test runner; the browser comes from `npm install` gets the test runner, and `npx playwright install --with-deps chromium` gets the
`npx playwright install --with-deps chromium` (in `install.sh`). browser.
## License ## License
MIT. See [LICENSE](LICENSE). MIT, see [LICENSE](LICENSE). The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by
@fontawesome, under [CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG.
The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by @fontawesome, under
[CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG in `web/index.html`.

30
TODO.md Normal file
View File

@@ -0,0 +1,30 @@
# To do
## Cut what is no longer needed
From a whole-repo audit for over-engineering on 2026-09-12. Biggest cut first.
- [x] **Pre-accounts adoption and the dead `entries` columns.** The copy of the old read state into
`entry_state` and the `entries.read`, `flagged` and `position` columns are gone. Its other half,
subscribing the first admin to the catalogue, was not dead and stays as `adopt_catalogue`.
(`src/db.rs`, `src/main.rs`)
- [x] **`contrib/` systemd units.** From before the container; nothing points at them.
- [x] **`migrate_opml_children`.** A one-time move of OPML children out of `config.toml` that has
run. Delete it and its call. (`src/main.rs`)
- [x] **The legacy `interval_mins` key.** Production uses `schedule`. Delete the field, the fallback
in `General::interval` and its test. (`src/config.rs`)
- [x] **`Db::entries` and `Db::count_entries`.** One-line wrappers only the tests call; the tests
call `entries_in` and `count_in` instead. (`src/db.rs`)
- [x] **`web::generate_token`.** Repeats `auth::new_session_token`. Use that. (`src/web.rs`)
- [x] **Page leftovers.** `globalEvery`, `S.busy`, `S.limit`, `unitOptions`' `firstLabel`, `--r`,
`.ep.open`, the phone `.ep .art`, the duplicate phone `.fhead.slim{flex-wrap}`, the second
`#sidebar{z-index}`, and the `on()` helper. (`web/index.html`)
- [x] **`logbuf` visitors.** `record_i64`, `record_u64` and `record_bool` repeat what `Visit`'s
defaults already do through `record_debug`. (`src/logbuf.rs`)
- [x] **The `infer` dependency.** Its torrent check is the `d8:announce` test on the next line.
- [x] **The `dirs` dependency.** `XDG_CONFIG_HOME`, `XDG_DATA_HOME` and `HOME` from `std::env`.
- [x] **The `tokio-stream` dependency.** `futures_util::stream::unfold` over the broadcast receiver.
- [x] **The icon inlined four times.** About 94 KB of base64 across both pages; serve it once as
`/icon.png` from `include_bytes!`, open without signing in like `/login`.
After these: `cargo test`, `node tests/page-smoke.js`, `npx playwright test`.

View File

@@ -1,7 +0,0 @@
[Unit]
Description=ipx feed scan (one shot)
[Service]
Type=oneshot
ExecStart=%h/.cargo/bin/ipx fetch
Environment=IPX_LOG=ipx=info

View File

@@ -1,18 +0,0 @@
# User unit: install to ~/.config/systemd/user/ipx.service, then
# systemctl --user enable --now ipx
# The socket lands in $XDG_RUNTIME_DIR/ipx.sock by default, so a UI running as the
# same user can attach without extra configuration.
[Unit]
Description=ipx podcatcher
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=%h/.cargo/bin/ipx daemon
Restart=on-failure
RestartSec=30
Environment=IPX_LOG=ipx=info
[Install]
WantedBy=default.target

View File

@@ -1,16 +0,0 @@
# Alternative to the daemon: a periodic one-shot scan, closer to how the original
# iPodderX agent was driven. Use this OR ipx.service, not both -- with no daemon
# running there is no socket, so a UI cannot attach.
#
# Install ipx-scan.service and ipx.timer to ~/.config/systemd/user/, then
# systemctl --user enable --now ipx.timer
[Unit]
Description=Periodic ipx feed scan
[Timer]
OnBootSec=5min
OnUnitActiveSec=1h
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -28,12 +28,15 @@ The page is compiled in, so **editing `web/index.html` needs a rebuild**.
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this). 1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there. 2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are 3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
re-derived into the database each scan, never written to config.toml. re-derived into the database each scan, never written to config.toml. A Patreon creator link
(a token, no `show=`) with more than one show is treated the same way, before any fetch: its
shows come from Patreon's web API and each becomes a derived feed.
4. Record entries. A changed title or description flips the item back to unread. 4. Record entries. A changed title or description flips the item back to unread.
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the 5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice. original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with 6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
a reason. a reason. What a filter skipped is judged again every scan, so a change of settings brings it
back. A feed in a group takes your settings on the group for anything you have not set on it.
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to 7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
or an error page — and is deleted. or an error page — and is deleted.
@@ -55,9 +58,9 @@ entry_state user_id, feed_id, guid, read, flagged, position
PK (user_id, feed_id, guid) PK (user_id, feed_id, guid)
``` ```
`entries` still has `read`, `flagged` and `position` columns from before accounts existed. They are Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
**dead** — the migration copied them into `entry_state` and nothing reads them now. Anything found before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
querying them is a bug; two were. older database.
Schema changes: add the table or column to `SCHEMA`, and for a column also to the list in Schema changes: add the table or column to `SCHEMA`, and for a column also to the list in
`migrate()`, which does `PRAGMA table_info` then `ALTER TABLE ADD COLUMN`. `Db::memory()` runs the `migrate()`, which does `PRAGMA table_info` then `ALTER TABLE ADD COLUMN`. `Db::memory()` runs the
@@ -108,7 +111,7 @@ else a `401`.
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) | | `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | | | `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) | | `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, starred, position | | `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, kept, position |
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning | | `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
| `POST /api/fetch` | | | `POST /api/fetch` | |
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML | | `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |

View File

@@ -60,7 +60,7 @@ ipx reap # actually delete
``` ```
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days` Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
with no file are pruned from the database. **Starred by anyone keeps a file**, and one only counts with no file are pruned from the database. **An item anyone kept keeps its file**, and one only counts
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
stops the next scan fetching it again. stops the next scan fetching it again.

View File

@@ -32,10 +32,10 @@ media_types = ["audio", "video"]
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI. be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`. * **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB * **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
pad. Starred items are never deleted, and a file only counts as read once every subscriber has pad. Kept items are never deleted, and a file only counts as read once every subscriber has
read it. `0` disables it entirely. read it. `0` disables it entirely.
* **`max_age_days`** — items older than this with no file on disk are pruned from the database. * **`max_age_days`** — items older than this with no file on disk are pruned from the database.
Starred ones stay. `0` disables it. Kept ones stay. `0` disables it.
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue. * **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap `0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
fetched 216 files and 22 GB in one scan. fetched 216 files and 22 GB in one scan.
@@ -43,8 +43,6 @@ media_types = ["audio", "video"]
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
without this the disk fills with artwork. Empty takes everything. without this the disk fills with artwork. Empty takes everything.
`interval_mins` from older configs is still read, and `schedule` supersedes it.
## `[torrent]` ## `[torrent]`
```toml ```toml

View File

@@ -6,6 +6,136 @@ reasoning lives. New write-ups go at the top.
See [README.md](../README.md) for what the thing is. See [README.md](../README.md) for what the thing is.
## 2026-09-12 — Cutting what had outlived its reason
A whole-repo audit for over-engineering listed twelve things to cut, and all of them went.
- **Upgrades from before accounts.** `migrate_opml_children` moved OPML feeds that old versions
wrote into `config.toml` out to the database, and ran at every daemon start to do nothing after
the first. Production ran it in 0.3.0; anything older has to pass through 0.4.0.
- **Half of the adoption, and not the other half.** The audit called `adopt_existing_library` a
one-time migration and it was cut whole. It did two jobs: copy the old read state into
`entry_state`, which was dead, and subscribe the first admin to the whole catalogue while nobody
subscribed to anything, which is how a fresh install's first account gets `config.toml`'s feeds.
The browser suite caught it at once, signing in to an empty sidebar; `cargo test` had no idea.
The second job is back as `adopt_catalogue`, with a unit test of its own.
- **The dead `entries` columns.** `read`, `flagged` and `position` moved to `entry_state` with
accounts. The adoption's copy was their last reader, but `record_entry` still wrote them, and
still reset `read` when a title changed, which nothing looked at. Two bugs came from queries
reading them. `migrate()` now drops them from an existing database (SQLite has had `DROP COLUMN`
since 3.35), and a test builds an old table to prove it. On production each drop rewrote the
66 MB `entries` table, about four seconds apiece, so the first start took thirteen. An
`ipx status` run in that window failed with `migrating schema`: every `ipx` command migrates when
it opens the database, and it collided with the daemon doing the same. A failed `ALTER TABLE`
changes nothing, and the database had been copied to `backup/` first anyway.
- **`interval_mins`**, which `schedule` replaced. An old config that still has the key loads; the
key is ignored, and the config test carries it to keep that true.
- **Three dependencies.** `infer` was only asked whether a file is a torrent, and the check after
it already looked for `d8:announce`, which is what `infer` looks for. `dirs` was three lookups of
`XDG_CONFIG_HOME`, `XDG_DATA_HOME` and `HOME`. `tokio-stream` wrapped the broadcast receiver for
the event stream; `futures_util::stream::unfold` does the same, lagging clients included.
- **Two token generators.** The web token came from a copy of the session-token code, with a
clock fallback on top. It uses `auth::new_session_token` now, and is 64 characters.
- **The icon inlined four times**, 23 KB of base64 each, into both pages. It is `/icon.png` now,
outside the sign-in wall with `/login`, since the sign-in page shows it.
- Also: the `contrib/` systemd units from before the container, `Db::entries` and
`Db::count_entries` that only the tests called, three `logbuf` visitors that repeated the trait's
defaults, and unused state, a helper and dead CSS in the page.
## 2026-09-11 — A design pass on the web UI
A review against screenshots of every view in all three themes found that Dark and Light read as a
generic dark dashboard: one pale blue did every job, most labels were bold, and nothing led. The
list it produced, in `TODO.md`, was worked through in one go. What was worth knowing:
- **Amber means new.** Badges, unread dots and download bars take the icon's EQ amber; blue is left
for the primary action and links. Light's amber was `#b06f10`, which gives white text 4.1:1,
short of AA for 11 px bold. It is `#9a5f0a` now, 5.2:1.
- **EQ bars mark what is playing.** Three `<i>` bars stand at 60, 100 and 40 % and animate only
while `body.playing` is set. The first version left the animation on but paused, expecting each
bar to hold a different frame. The frames it held were within a pixel of each other, and on
screen the bars read as three dots. Under reduced motion one rule drops every animation and
transition, which leaves the bars standing.
- **The focus ring was clipped.** `.tgroup` and `#topbar` both set `overflow:hidden`, so a ring
drawn outside a toolbar button was cut off. Rings inside clipping parents are inset instead.
- **The feed list could not be used from the keyboard at all.** Rows were `<div>`s and the triangle
a `<span>`, so Tab went from the feed filter to Sign out. Rows now take focus, the triangle is a
`<button aria-expanded>`, and `renderFeeds` puts focus back on the same row after redrawing,
since every live update replaces every row. The list's own key handler stops Space and the
arrows from reaching the player's shortcuts on the document.
- **The triangle hangs in the margin.** Every row used to reserve an 18 px slot for it, pushing a
hundred feeds 28 px right of the places above for the sake of two folders. It is now absolutely
placed in the row's left padding, the full height of the row, so a near miss no longer opens
the folder's page.
- **A selected tile vanished** because the initials tile and the selected row were both `--raise`.
Tiles now mix their tint into `--bg`, which no row uses.
- **The Files pane hides itself** with `#split:has(>#files[hidden])`, which collapses its column.
The phone layout already hides the pane, and a zero-width extra track there is harmless.
- **Flagged became Kept** in the tab, and in the server's refusal to delete a file someone else
kept. The filter value and the column stay `flagged`; renaming those buys nothing.
## 2026-09-11 — A Patreon creator is a list of shows
Ray asked whether ipx could sync with Patreon. Not in full. The documented API (v2, the
`identity.memberships` scope) lists the creators you back and whether each has a feed (`has_rss`),
but no resource carries the `auth` token that makes a feed URL work. That token only comes from the
creator's page. It is also one per membership, not one per account: techpod's differs from Glass
Cannon's, so no single token finds everything you back.
What does work is one creator at a time, which is what Ray wanted for Glass Cannon and its 33 shows:
- `patreon.com/rss?auth=<token>`, with no creator named, returns that token's creator. Its self link,
about 660 bytes in, gives the campaign by number (`/rss/369921`). Patreon ignores `Range` here, so
ipx reads the stream until the number appears and hangs up, instead of taking all 2.8 MB.
- A show's `show=` number is a Patreon collection. Asked anonymously, the collection listing
(`/api/collection?filter[campaign_id]=`) and a post's `collections` both hide patron-only ones: you
get "FAQ". `/api/campaigns/<id>?include=shows` lists every show, anonymously, in one response.
- Every spelling works: `rss/glasscannon?auth=…&show=N`, `rss/369921?…` and `rss?auth=…&show=N` all
serve the same 131 items. Enclosure URLs are the same in the creator feed and the show feed, and
stable between fetches.
That last point shaped the design. `enclosures.url` is unique, so whichever feed is scanned first owns
the file. The first cut only asked a creator for its shows while it had no entries of its own, so that a
creator already read as a plain feed, holding every show's episodes, would never be split into shows
that came up empty. Within the hour that was the wrong call: Glass Cannon had gone into production on
the build before this one, been read as one feed of 2,385 items, and the rule kept it that way. Finding
anything in that heap was the problem Ray wanted solved.
So a creator with more than one show is always a group, run through the same sync as an OPML
(`sync_group`, split out of `sync_opml`). When it becomes one, its items are cleared and each show
takes over the enclosures the creator holds as the show lists them (`Db::adopt`), downloaded files and
everyone's read state included. One show leaves it a plain feed, which is what techpod already was. If
the shows cannot be listed, a creator already split fails the scan rather than being read as one heap;
one that never was is read as one feed until they can be. An answer without a `shows` list is an error,
not "no shows".
**Filter verdicts follow the settings.** Ray also reported that turning on Allow explicit and
rescanning brought nothing back. An item was judged once, when first seen, and `skipped` was final. The
2026-09-10 entry below saw it coming ("worth a `ipx retry <feed>` command if this bites"). It bit:
2,166 Glass Cannon items and all 88 of Shadowdark's sat at `skipped: explicit` with the setting on.
Every scan now runs the filters again over what they skipped (not over `torrents disabled`, which is not
a filter's call) and requeues what they now let through. Only that direction: a queued item is never
pulled back, because Download latest and a manual download both work by queueing.
**Two gaps beside it.** Add feed had no explicit box, so every new feed's first scan skipped all its
explicit items; it has one now, stored on your subscription like the feed dialog's. And a feed inside a
group ignored your settings on the group, though the group's dialog said they were inherited: settings
live on each person's subscription, and nothing read the group's. `Db::subscribers` now fills what you
have not set on the feed from your subscription to the group, and the feed list shows the same.
**A name that was already used.** Replaying the split on a copy of the production database left one
show with a Supercast episode in it. "Glass Cannon Live! Ascension | Pathfinder 2E" slugs to
`glass-cannon-live-ascension-pathfinder-2`, the id of a Supercast feed of the same show that had been
removed. Removing a feed keeps its rows on purpose, so that re-adding it does not fetch the back
catalogue again, but choosing a new id only checked config.toml and derived feeds. The Patreon show took
the old id and everything still filed under it. An id is now also taken when the database has a feed by
that id at a different URL; the same URL may still have it back, which is the re-add case.
Shows already added by hand are matched by token and show number, not by exact URL (`same_feed`), so a
bare token does not add Get in the Trunk and Shadowdark a second time under another spelling.
The show listing is Patreon's own undocumented web API. If it changes, only finding new shows stops.
## 2026-09-11 — One meaning per icon, sortable columns, and one player ## 2026-09-11 — One meaning per icon, sortable columns, and one player
Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act

View File

@@ -8,7 +8,7 @@ fetch, one parse and one file.
| Yours alone | The same for everyone | | Yours alone | The same for everyone |
|---|---| |---|---|
| Read, starred, playback position | The feed's URL | | Read, kept, playback position | The feed's URL |
| Which feeds you see at all | Its download folder | | Which feeds you see at all | Its download folder |
| Keywords, auto-download, explicit, per-scan cap | When it is scanned | | Keywords, auto-download, explicit, per-scan cap | When it is scanned |
| | The file on disk | | | The file on disk |
@@ -34,10 +34,10 @@ is shared.
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
for everyone** and names them in the confirmation, and the server has the last word: if anyone else for everyone** and names them in the confirmation, and the server has the last word: if anyone else
has starred the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the has kept the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
reason, and only `?force=true` goes through. reason, and only `?force=true` goes through.
Retention follows the same rule: starred by anyone keeps a file, and it counts as read only once Retention follows the same rule: an item anyone kept keeps its file, and it counts as read only once
every subscriber has read it. every subscriber has read it.
## Signing in ## Signing in

View File

@@ -26,9 +26,6 @@ pub struct General {
/// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d". /// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d".
/// A feed's own `schedule` overrides this. /// A feed's own `schedule` overrides this.
pub schedule: String, pub schedule: String,
/// Superseded by `schedule`. Still read so existing configs keep working.
#[serde(skip_serializing_if = "Option::is_none")]
pub interval_mins: Option<u64>,
pub organize: Organize, pub organize: Organize,
/// 0 = unlimited. /// 0 = unlimited.
pub max_total_gb: f64, pub max_total_gb: f64,
@@ -153,7 +150,6 @@ impl Default for General {
download_dir: home().join("Podcasts"), download_dir: home().join("Podcasts"),
socket: default_socket(), socket: default_socket(),
schedule: "every 60m".into(), schedule: "every 60m".into(),
interval_mins: None,
organize: Organize::Feed, organize: Organize::Feed,
max_total_gb: 0.0, max_total_gb: 0.0,
max_age_days: 0, max_age_days: 0,
@@ -176,8 +172,8 @@ impl Default for Torrent {
} }
impl General { impl General {
/// Minutes between checks. Falls back to the legacy `interval_mins`, then to an hour. /// Minutes between checks, or an hour when `schedule` is empty or unreadable. A malformed
/// A malformed value warns rather than stopping the daemon. /// value warns rather than stopping the daemon.
pub fn interval(&self) -> u64 { pub fn interval(&self) -> u64 {
if let Some(n) = parse_interval(&self.schedule) { if let Some(n) = parse_interval(&self.schedule) {
return n; return n;
@@ -185,7 +181,7 @@ impl General {
if !self.schedule.trim().is_empty() { if !self.schedule.trim().is_empty() {
tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default"); tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default");
} }
self.interval_mins.filter(|n| *n > 0).unwrap_or(60) 60
} }
} }
@@ -278,9 +274,7 @@ pub fn config_path() -> PathBuf {
if let Ok(p) = std::env::var("IPX_CONFIG") { if let Ok(p) = std::env::var("IPX_CONFIG") {
return PathBuf::from(p); return PathBuf::from(p);
} }
dirs::config_dir() xdg("XDG_CONFIG_HOME", ".config").join("ipx/config.toml")
.unwrap_or_else(|| home().join(".config"))
.join("ipx/config.toml")
} }
/// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`. /// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`.
@@ -288,9 +282,7 @@ pub fn data_dir() -> PathBuf {
if let Ok(p) = std::env::var("IPX_DATA_DIR") { if let Ok(p) = std::env::var("IPX_DATA_DIR") {
return PathBuf::from(p); return PathBuf::from(p);
} }
dirs::data_dir() xdg("XDG_DATA_HOME", ".local/share").join("ipx")
.unwrap_or_else(|| home().join(".local/share"))
.join("ipx")
} }
fn default_socket() -> PathBuf { fn default_socket() -> PathBuf {
@@ -346,8 +338,16 @@ pub fn unique_slug(text: &str, taken: &BTreeMap<String, Feed>) -> String {
(2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap() (2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap()
} }
/// `$var`, or `~/fallback` when it is unset or empty, as the XDG base directory spec says.
fn xdg(var: &str, fallback: &str) -> PathBuf {
std::env::var_os(var)
.filter(|v| !v.is_empty())
.map(PathBuf::from)
.unwrap_or_else(|| home().join(fallback))
}
fn home() -> PathBuf { fn home() -> PathBuf {
dirs::home_dir().unwrap_or_else(|| PathBuf::from(".")) std::env::var_os("HOME").map(PathBuf::from).unwrap_or_else(|| PathBuf::from("."))
} }
fn expand_tilde(p: &Path) -> PathBuf { fn expand_tilde(p: &Path) -> PathBuf {
@@ -367,6 +367,8 @@ mod tests {
r#" r#"
[general] [general]
download_dir = "/tmp/pods" download_dir = "/tmp/pods"
# A key older versions read. An old config that still has it has to load.
interval_mins = 45
[feeds.example] [feeds.example]
url = "https://example.com/feed.xml" url = "https://example.com/feed.xml"
@@ -403,22 +405,17 @@ mod tests {
} }
#[test] #[test]
fn interval_falls_back_through_legacy_then_default() { fn interval_falls_back_to_an_hour() {
let mut g = General::default(); let mut g = General::default();
assert_eq!(g.interval(), 60, "the default schedule"); assert_eq!(g.interval(), 60, "the default schedule");
g.schedule = "every 15m".into(); g.schedule = "every 15m".into();
assert_eq!(g.interval(), 15); assert_eq!(g.interval(), 15);
// A config written before `schedule` existed still works. // Empty or garbage must not stop the daemon.
g.schedule = String::new(); g.schedule = String::new();
g.interval_mins = Some(45); assert_eq!(g.interval(), 60);
assert_eq!(g.interval(), 45);
// Garbage must not stop the daemon.
g.schedule = "whenever".into(); g.schedule = "whenever".into();
assert_eq!(g.interval(), 45);
g.interval_mins = None;
assert_eq!(g.interval(), 60); assert_eq!(g.interval(), 60);
} }

292
src/db.rs
View File

@@ -40,14 +40,10 @@ CREATE TABLE IF NOT EXISTS entries (
published INTEGER, published INTEGER,
description TEXT, description TEXT,
first_seen INTEGER NOT NULL, first_seen INTEGER NOT NULL,
read INTEGER NOT NULL DEFAULT 0,
flagged INTEGER NOT NULL DEFAULT 0,
image TEXT, image TEXT,
duration INTEGER, duration INTEGER,
episode INTEGER, episode INTEGER,
season INTEGER, season INTEGER,
-- Seconds into the audio, so playback resumes where it was left.
position INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (feed_id, guid) PRIMARY KEY (feed_id, guid)
); );
@@ -93,7 +89,7 @@ CREATE TABLE IF NOT EXISTS subscriptions (
PRIMARY KEY (user_id, feed_id) PRIMARY KEY (user_id, feed_id)
); );
-- Read, starred and how far in. One row per person per item, created on first touch; -- Read, kept and how far in. One row per person per item, created on first touch;
-- an item nobody has touched has no row at all, which is what unread means. -- an item nobody has touched has no row at all, which is what unread means.
CREATE TABLE IF NOT EXISTS entry_state ( CREATE TABLE IF NOT EXISTS entry_state (
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
@@ -143,8 +139,8 @@ pub struct Managed {
pub orphaned: bool, pub orphaned: bool,
} }
/// Adds columns that later versions introduced. CREATE TABLE IF NOT EXISTS does nothing to /// Adds the columns later versions introduced and drops the ones they retired. CREATE TABLE IF
/// a table that already exists, so an installed database needs them added explicitly. /// NOT EXISTS does nothing to a table that already exists, so an installed database needs both.
fn migrate(conn: &Connection) -> Result<()> { fn migrate(conn: &Connection) -> Result<()> {
let wanted: &[(&str, &str, &str)] = &[ let wanted: &[(&str, &str, &str)] = &[
("feeds", "image", "TEXT"), ("feeds", "image", "TEXT"),
@@ -155,18 +151,29 @@ fn migrate(conn: &Connection) -> Result<()> {
("entries", "duration", "INTEGER"), ("entries", "duration", "INTEGER"),
("entries", "episode", "INTEGER"), ("entries", "episode", "INTEGER"),
("entries", "season", "INTEGER"), ("entries", "season", "INTEGER"),
("entries", "position", "INTEGER NOT NULL DEFAULT 0"),
]; ];
for (table, column, ty) in wanted { // Read state from before accounts, long since moved to entry_state. Two bugs came from
// queries still reading these after they stopped meaning anything, so they go.
let retired: &[(&str, &str)] = &[("entries", "read"), ("entries", "flagged"), ("entries", "position")];
let has = |table: &str, column: &str| -> Result<bool> {
let mut stmt = conn.prepare(&format!("PRAGMA table_info({table})"))?; let mut stmt = conn.prepare(&format!("PRAGMA table_info({table})"))?;
let existing: Vec<String> = stmt let names = stmt
.query_map([], |r| r.get::<_, String>(1))? .query_map([], |r| r.get::<_, String>(1))?
.collect::<rusqlite::Result<Vec<_>>>()?; .collect::<rusqlite::Result<Vec<_>>>()?;
if !existing.iter().any(|c| c == column) { Ok(names.iter().any(|c| c == column))
};
for (table, column, ty) in wanted {
if !has(table, column)? {
tracing::info!(table, column, "adding column"); tracing::info!(table, column, "adding column");
conn.execute_batch(&format!("ALTER TABLE {table} ADD COLUMN {column} {ty}"))?; conn.execute_batch(&format!("ALTER TABLE {table} ADD COLUMN {column} {ty}"))?;
} }
} }
for (table, column) in retired {
if has(table, column)? {
tracing::info!(table, column, "dropping column");
conn.execute_batch(&format!("ALTER TABLE {table} DROP COLUMN {column}"))?;
}
}
Ok(()) Ok(())
} }
@@ -345,22 +352,19 @@ impl Db {
/// Returns true when this entry had not been seen before. /// Returns true when this entry had not been seen before.
/// ///
/// A changed description or title flips `read` back to 0, which is what the original's
/// textDiff dance was ultimately for -- minus the diff markup, which the UI can do.
pub fn record_entry(&self, feed_id: &str, e: &crate::feed::Entry) -> Result<bool> { pub fn record_entry(&self, feed_id: &str, e: &crate::feed::Entry) -> Result<bool> {
let conn = self.conn.lock().unwrap(); let conn = self.conn.lock().unwrap();
let inserted = conn.execute( let inserted = conn.execute(
"INSERT OR IGNORE INTO entries "INSERT OR IGNORE INTO entries
(feed_id, guid, title, link, published, description, first_seen, read, flagged, (feed_id, guid, title, link, published, description, first_seen,
image, duration, episode, season) image, duration, episode, season)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0, 0, ?8, ?9, ?10, ?11)", VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)",
rusqlite::params![ rusqlite::params![
feed_id, e.guid, e.title, e.link, e.published, e.description, now(), feed_id, e.guid, e.title, e.link, e.published, e.description, now(),
e.image, e.duration, e.episode, e.season e.image, e.duration, e.episode, e.season
], ],
)?; )?;
if inserted == 0 { if inserted == 0 {
// The SET expressions see the pre-update row, so this compares old vs new.
conn.execute( conn.execute(
"UPDATE entries SET "UPDATE entries SET
title = coalesce(?3, title), title = coalesce(?3, title),
@@ -368,8 +372,7 @@ impl Db {
image = coalesce(?5, image), image = coalesce(?5, image),
duration = coalesce(?6, duration), duration = coalesce(?6, duration),
episode = coalesce(?7, episode), episode = coalesce(?7, episode),
season = coalesce(?8, season), season = coalesce(?8, season)
read = CASE WHEN description IS NOT ?4 OR title IS NOT ?3 THEN 0 ELSE read END
WHERE feed_id = ?1 AND guid = ?2", WHERE feed_id = ?1 AND guid = ?2",
rusqlite::params![ rusqlite::params![
feed_id, e.guid, e.title, e.description, feed_id, e.guid, e.title, e.description,
@@ -687,22 +690,9 @@ pub struct EncRow {
} }
impl Db { impl Db {
/// One page of a feed's entries, newest first, each with its enclosures attached. /// One page of entries, each with its enclosures attached: one feed's, or every feed the
/// `search` matches title and description, case-insensitively. /// person subscribes to when `feed_id` is None (All Subscriptions). `search` matches title
pub fn entries( /// and description, case-insensitively.
&self,
user_id: i64,
feed_id: &str,
filter: Filter,
search: Option<&str>,
offset: i64,
limit: i64,
) -> Result<Vec<EntryRow>> {
self.entries_in(user_id, Some(feed_id), filter, search, offset, limit, &order_sql("published", "desc"))
}
/// `entries` for one feed, or across every feed the person subscribes to when `feed_id`
/// is None: the All Subscriptions view.
pub fn entries_in( pub fn entries_in(
&self, &self,
user_id: i64, user_id: i64,
@@ -793,18 +783,7 @@ impl Db {
Ok(rows) Ok(rows)
} }
/// How many entries match, so the UI knows whether there is another page. /// How many entries `entries_in` would page through, so the UI knows whether there is more.
pub fn count_entries(
&self,
user_id: i64,
feed_id: &str,
filter: Filter,
search: Option<&str>,
) -> Result<i64> {
self.count_in(user_id, Some(feed_id), filter, search)
}
/// `count_entries` for one feed, or across every feed the person subscribes to.
pub fn count_in( pub fn count_in(
&self, &self,
user_id: i64, user_id: i64,
@@ -838,24 +817,16 @@ impl Db {
Ok(()) Ok(())
} }
/// Marks every entry in a feed read, for the "mark all read" button. /// The first admin starts subscribed to the whole catalogue: whoever wrote config.toml meant
/// Moves a single-user library onto an account: everything read, starred or part-played /// to read those feeds, and without this a fresh install signs in to an empty sidebar. Runs
/// becomes that person's, and they subscribe to every feed already in the catalogue. /// only while nobody subscribes to anything, so an unsubscribe is never undone.
/// Runs once -- the moment there is a first account and no subscriptions yet. pub fn adopt_catalogue(&self, user_id: i64, catalogue: &[String]) -> Result<usize> {
pub fn adopt_existing_library(&self, user_id: i64, catalogue: &[String]) -> Result<usize> {
let conn = self.conn.lock().unwrap(); let conn = self.conn.lock().unwrap();
let already: i64 = let already: i64 =
conn.query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0))?; conn.query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0))?;
if already > 0 { if already > 0 {
return Ok(0); return Ok(0);
} }
let moved = conn.execute(
"INSERT INTO entry_state (user_id, feed_id, guid, read, flagged, position)
SELECT ?1, feed_id, guid, read, flagged, position FROM entries
WHERE read = 1 OR flagged = 1 OR position > 0
ON CONFLICT(user_id, feed_id, guid) DO NOTHING",
[user_id],
)?;
for id in catalogue { for id in catalogue {
conn.execute( conn.execute(
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created) VALUES (?1, ?2, ?3)", "INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created) VALUES (?1, ?2, ?3)",
@@ -868,7 +839,7 @@ impl Db {
SELECT ?1, id, ?2 FROM feeds", SELECT ?1, id, ?2 FROM feeds",
params![user_id, now()], params![user_id, now()],
)?; )?;
Ok(moved) Ok(catalogue.len())
} }
// ---- subscriptions ---- // ---- subscriptions ----
@@ -920,14 +891,20 @@ impl Db {
/// Everyone's settings for one feed. The scanner merges these into what it fetches /// Everyone's settings for one feed. The scanner merges these into what it fetches
/// and downloads, since one file serves the lot. /// and downloads, since one file serves the lot.
pub fn subscribers(&self, feed_id: &str) -> Result<Vec<Sub>> { /// In a group, whatever someone has not set on the feed itself comes from their
/// subscription to the group, as the group's settings dialog has always said it does.
pub fn subscribers(&self, feed_id: &str, group: Option<&str>) -> Result<Vec<Sub>> {
let conn = self.conn.lock().unwrap(); let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT keywords, auto_download, allow_explicit, max_new_per_check "SELECT coalesce(c.keywords, p.keywords), coalesce(c.auto_download, p.auto_download),
FROM subscriptions WHERE feed_id = ?1", coalesce(c.allow_explicit, p.allow_explicit),
coalesce(c.max_new_per_check, p.max_new_per_check)
FROM subscriptions c
LEFT JOIN subscriptions p ON p.user_id = c.user_id AND p.feed_id = ?2
WHERE c.feed_id = ?1",
)?; )?;
let out = stmt let out = stmt
.query_map([feed_id], |r| { .query_map(params![feed_id, group], |r| {
Ok(Sub { Ok(Sub {
feed_id: feed_id.to_string(), feed_id: feed_id.to_string(),
keywords: r keywords: r
@@ -1325,6 +1302,71 @@ impl Db {
Ok(()) Ok(())
} }
/// Empties a feed of its items, leaving its files alone.
pub fn clear_entries(&self, feed_id: &str) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute("DELETE FROM entries WHERE feed_id = ?1", [feed_id])?;
Ok(())
}
/// Every feed the database holds rows for, as (id, url), removed ones included.
pub fn feed_urls(&self) -> Result<Vec<(String, String)>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare("SELECT id, coalesce(url, '') FROM feeds")?;
let out = stmt
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))?
.collect::<rusqlite::Result<_>>()?;
Ok(out)
}
/// Hands a feed in a group the enclosures its parent holds, as (guid, url), with everyone's
/// read state for them. A Patreon creator read as one feed before it was split into shows
/// owns every show's files, and `enclosures.url` is unique, so without this each show would
/// list its items with nothing to play.
pub fn adopt(&self, parent: &str, child: &str, listed: &[(&str, &str)]) -> Result<()> {
let mut conn = self.conn.lock().unwrap();
let holds: bool = conn.query_row(
"SELECT EXISTS (SELECT 1 FROM enclosures WHERE feed_id = ?1)",
[parent],
|r| r.get(0),
)?;
if !holds {
return Ok(()); // An OPML, or a creator already shared out.
}
let tx = conn.transaction()?;
for &(guid, url) in listed {
let moved = tx.execute(
"UPDATE enclosures SET feed_id = ?3, guid = ?4 WHERE url = ?1 AND feed_id = ?2",
params![url, parent, child, guid],
)?;
if moved == 1 {
// Patreon gives a post the same guid in every feed it appears in.
tx.execute(
"UPDATE OR IGNORE entry_state SET feed_id = ?2 WHERE feed_id = ?1 AND guid = ?3",
params![parent, child, guid],
)?;
}
}
tx.commit()?;
Ok(())
}
/// A feed's enclosures skipped by one of its filters, by URL, with the reason: the verdicts a
/// change of settings can overturn. A torrent held back while torrents are off is not a
/// filter's call.
pub fn skipped_by_filter(&self, feed_id: &str) -> Result<std::collections::HashMap<String, String>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare(
"SELECT url, last_error FROM enclosures
WHERE feed_id = ?1 AND state = 'skipped' AND last_error IS NOT NULL
AND last_error != 'torrents disabled'",
)?;
let out = stmt
.query_map([feed_id], |r| Ok((r.get(0)?, r.get(1)?)))?
.collect::<rusqlite::Result<_>>()?;
Ok(out)
}
pub fn set_orphaned(&self, feed_id: &str, on: bool) -> Result<()> { pub fn set_orphaned(&self, feed_id: &str, on: bool) -> Result<()> {
let conn = self.conn.lock().unwrap(); let conn = self.conn.lock().unwrap();
conn.execute( conn.execute(
@@ -1457,8 +1499,9 @@ mod tests {
// Starring and position are just as private. // Starring and position are just as private.
db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).unwrap(); db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).unwrap();
db.set_position(2, "f", "b", 42).unwrap(); db.set_position(2, "f", "b", 42).unwrap();
let ray = db.entries(1, "f", Filter::All, None, 0, 50).unwrap(); let order = order_sql("published", "desc");
let sam = db.entries(2, "f", Filter::All, None, 0, 50).unwrap(); let page = |user| db.entries_in(user, Some("f"), Filter::All, None, 0, 50, &order).unwrap();
let (ray, sam) = (page(1), page(2));
let ray_b = ray.iter().find(|e| e.guid == "b").unwrap(); let ray_b = ray.iter().find(|e| e.guid == "b").unwrap();
let sam_b = sam.iter().find(|e| e.guid == "b").unwrap(); let sam_b = sam.iter().find(|e| e.guid == "b").unwrap();
assert!(ray_b.flagged && ray_b.position == 0); assert!(ray_b.flagged && ray_b.position == 0);
@@ -1482,6 +1525,49 @@ mod tests {
assert_eq!(sum.downloaded, 0); assert_eq!(sum.downloaded, 0);
} }
#[test]
fn an_old_database_loses_the_retired_read_columns() {
let conn = Connection::open_in_memory().unwrap();
conn.execute_batch(
"CREATE TABLE entries (feed_id TEXT NOT NULL, guid TEXT NOT NULL,
first_seen INTEGER NOT NULL, read INTEGER NOT NULL DEFAULT 0,
flagged INTEGER NOT NULL DEFAULT 0, position INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (feed_id, guid));",
)
.unwrap();
// The same order as open(): the schema leaves the old table alone, migrate() fixes it.
conn.execute_batch(SCHEMA).unwrap();
migrate(&conn).unwrap();
let cols: Vec<String> = conn
.prepare("PRAGMA table_info(entries)")
.unwrap()
.query_map([], |r| r.get(1))
.unwrap()
.collect::<rusqlite::Result<_>>()
.unwrap();
assert!(!cols.iter().any(|c| ["read", "flagged", "position"].contains(&c.as_str())), "{cols:?}");
assert!(cols.iter().any(|c| c == "image"), "and it still gains the newer ones");
}
#[test]
fn the_first_admin_starts_with_the_catalogue_and_only_once() {
// Cutting this along with the dead read columns left the browser suite's admin with an
// empty sidebar: it is how a fresh install's first account gets config.toml's feeds.
let db = Db::memory().unwrap();
db.exec_for_test("INSERT INTO users (id, name, is_admin, created) VALUES (1,'admin',1,0);")
.unwrap();
let subs = || -> i64 {
db.conn.lock().unwrap().query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0)).unwrap()
};
let catalogue = ["a".to_string(), "b".to_string()];
assert_eq!(db.adopt_catalogue(1, &catalogue).unwrap(), 2);
assert_eq!(subs(), 2);
// Once anyone subscribes to anything it never runs again, so an unsubscribe sticks.
db.unsubscribe(1, "a").unwrap();
assert_eq!(db.adopt_catalogue(1, &catalogue).unwrap(), 0);
assert_eq!(subs(), 1);
}
#[test] #[test]
fn every_filter_works_with_and_without_a_search_term() { fn every_filter_works_with_and_without_a_search_term() {
// Regression: the search clause used to be omitted when no term was given, while // Regression: the search clause used to be omitted when no term was given, while
@@ -1505,21 +1591,22 @@ mod tests {
for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged] { for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged] {
// Both paths must run without erroring, and agree with each other. // Both paths must run without erroring, and agree with each other.
let rows = db.entries(7, "f", f, None, 0, 50).unwrap(); let order = order_sql("published", "desc");
let n = db.count_entries(7, "f", f, None).unwrap(); let rows = db.entries_in(7, Some("f"), f, None, 0, 50, &order).unwrap();
let n = db.count_in(7, Some("f"), f, None).unwrap();
assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page"); assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page");
let rows = db.entries(7, "f", f, Some("dive"), 0, 50).unwrap(); let rows = db.entries_in(7, Some("f"), f, Some("dive"), 0, 50, &order).unwrap();
let n = db.count_entries(7, "f", f, Some("dive")).unwrap(); let n = db.count_in(7, Some("f"), f, Some("dive")).unwrap();
assert_eq!(rows.len() as i64, n, "{f:?} with search disagrees"); assert_eq!(rows.len() as i64, n, "{f:?} with search disagrees");
} }
assert_eq!(db.count_entries(7, "f", Filter::All, None).unwrap(), 3); assert_eq!(db.count_in(7, Some("f"), Filter::All, None).unwrap(), 3);
assert_eq!(db.count_entries(7, "f", Filter::Unread, None).unwrap(), 1); assert_eq!(db.count_in(7, Some("f"), Filter::Unread, None).unwrap(), 1);
assert_eq!(db.count_entries(7, "f", Filter::Downloaded, None).unwrap(), 1); assert_eq!(db.count_in(7, Some("f"), Filter::Downloaded, None).unwrap(), 1);
assert_eq!(db.count_entries(7, "f", Filter::Flagged, None).unwrap(), 1); assert_eq!(db.count_in(7, Some("f"), Filter::Flagged, None).unwrap(), 1);
assert_eq!(db.count_entries(7, "f", Filter::All, Some("dive")).unwrap(), 2); assert_eq!(db.count_in(7, Some("f"), Filter::All, Some("dive")).unwrap(), 2);
assert_eq!(db.count_entries(7, "f", Filter::All, Some("NOTES two")).unwrap(), 1, assert_eq!(db.count_in(7, Some("f"), Filter::All, Some("NOTES two")).unwrap(), 1,
"search is case-insensitive and covers the description"); "search is case-insensitive and covers the description");
} }
@@ -1561,6 +1648,61 @@ mod tests {
assert_eq!(state(4), "done"); assert_eq!(state(4), "done");
} }
#[test]
fn a_show_takes_over_what_its_creator_held() {
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0);
INSERT INTO enclosures (id, feed_id, guid, url, state, path, last_error) VALUES
(1,'creator','a','u1','done','/x/a.mp3',NULL),
(2,'creator','b','u2','skipped',NULL,'explicit'),
(3,'creator','c','u3','skipped',NULL,'explicit'),
(4,'other','d','u4','skipped',NULL,'torrents disabled');
INSERT INTO entry_state (user_id, feed_id, guid, read) VALUES (1,'creator','a',1);",
)
.unwrap();
db.adopt("creator", "show", &[("a", "u1"), ("b", "u2"), ("d", "u4")]).unwrap();
{
let conn = db.conn.lock().unwrap();
let owner = |id: i64| -> String {
conn.query_row("SELECT feed_id FROM enclosures WHERE id = ?1", [id], |r| r.get(0)).unwrap()
};
assert_eq!(owner(1), "show", "a downloaded file moves with its item");
assert_eq!(owner(2), "show");
assert_eq!(owner(3), "creator", "this show does not list it");
assert_eq!(owner(4), "other", "only the parent's are taken");
let read: String = conn
.query_row("SELECT feed_id FROM entry_state WHERE user_id = 1 AND guid = 'a'", [], |r| r.get(0))
.unwrap();
assert_eq!(read, "show", "what you had read stays read");
}
// Only a filter's verdict can be overturned by a change of settings.
let skipped = db.skipped_by_filter("show").unwrap();
assert_eq!(skipped.get("u2").map(String::as_str), Some("explicit"));
assert_eq!(skipped.len(), 1);
assert!(db.skipped_by_filter("other").unwrap().is_empty(), "torrents disabled is not a filter");
}
#[test]
fn a_feed_in_a_group_follows_your_settings_on_the_group() {
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0);
INSERT INTO subscriptions (user_id, feed_id, allow_explicit, created) VALUES
(1,'group',1,0),(1,'show',NULL,0),(2,'group',1,0),(2,'show',0,0);",
)
.unwrap();
let explicit = |group| -> Vec<Option<bool>> {
let mut v: Vec<_> =
db.subscribers("show", group).unwrap().into_iter().map(|s| s.allow_explicit).collect();
v.sort();
v
};
assert_eq!(explicit(Some("group")), [Some(false), Some(true)], "ray inherits; sam's own choice on the show wins");
assert_eq!(explicit(None), [None, Some(false)], "outside a group nothing is inherited");
}
#[test] #[test]
fn enclosure_url_is_the_dedupe_key() { fn enclosure_url_is_the_dedupe_key() {
let db = Db::memory().unwrap(); let db = Db::memory().unwrap();

View File

@@ -223,7 +223,7 @@ enum Sniffed {
/// 2008 and so always answered 'data'. /// 2008 and so always answered 'data'.
async fn sniff(path: &Path) -> Result<Sniffed> { async fn sniff(path: &Path) -> Result<Sniffed> {
let head = read_head(path, 512).await?; let head = read_head(path, 512).await?;
if infer::is(&head, "torrent") || head.starts_with(b"d8:announce") || head.starts_with(b"d7:") { if head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
return Ok(Sniffed::Torrent); return Ok(Sniffed::Torrent);
} }
let text = String::from_utf8_lossy(&head); let text = String::from_utf8_lossy(&head);

View File

@@ -117,6 +117,103 @@ pub fn opml_title(bytes: &[u8]) -> Option<String> {
.filter(|t| !t.is_empty()) .filter(|t| !t.is_empty())
} }
/// The token and show of a Patreon feed link, or None for any other URL.
///
/// Patreon gives each patron one token per creator. With no show it stands for the creator,
/// whose feed carries every show at once.
fn patreon_parts(url: &str) -> Option<(String, Option<String>)> {
let u = url::Url::parse(url).ok()?;
if !matches!(u.host_str()?, "patreon.com" | "www.patreon.com") || !u.path().starts_with("/rss") {
return None;
}
let param = |name: &str| u.query_pairs().find(|(k, _)| k == name).map(|(_, v)| v.into_owned());
Some((param("auth")?, param("show")))
}
/// A Patreon link naming a creator but no show.
pub fn is_patreon_creator(url: &str) -> bool {
matches!(patreon_parts(url), Some((_, None)))
}
/// What was typed into Add feed, as a URL. A bare Patreon token is taken as its creator's
/// feed, since the token alone says whose it is.
pub fn expand_input(input: &str) -> String {
let s = input.trim();
let token = s.len() >= 20 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_');
if token { format!("https://www.patreon.com/rss?auth={s}") } else { s.to_owned() }
}
/// Whether two URLs are the same feed. One Patreon show has several spellings -- by the
/// creator's name, by number, or with no creator at all -- and the token and show are what
/// identify it.
pub fn same_feed(a: &str, b: &str) -> bool {
a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b))
}
/// A Patreon creator's name and shows, each show as (title, feed URL).
///
/// ponytail: Patreon's own web API, undocumented, asked without signing in. If it changes,
/// finding shows stops and the show feeds already found keep working. The documented API
/// needs an OAuth client per install and does not list shows.
pub async fn patreon_shows(
client: &reqwest::Client,
url: &str,
) -> Result<(Option<String>, Vec<(String, String)>)> {
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
let mut resp = client.get(url).send().await.context("connecting")?;
if !resp.status().is_success() {
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
}
let mut head = Vec::new();
while patreon_campaign(&head).is_none() && head.len() < 64 * 1024 {
let Some(chunk) = resp.chunk().await.context("reading the feed")? else { break };
head.extend_from_slice(&chunk);
}
let campaign = patreon_campaign(&head)
.ok_or_else(|| anyhow!("the Patreon feed does not say whose it is"))?;
let api = format!(
"https://www.patreon.com/api/campaigns/{campaign}\
?include=shows&fields%5Bcampaign%5D=name&fields%5Bcollection%5D=title"
);
let resp = client.get(api).send().await.context("asking Patreon for the shows")?;
if !resp.status().is_success() {
return Err(anyhow!("Patreon would not list the shows: HTTP {}", resp.status()));
}
let (name, shows) = parse_patreon_shows(&resp.bytes().await.context("reading the shows")?)?;
Ok((name, shows.into_iter().map(|(id, title)| (title, format!("{url}&show={id}"))).collect()))
}
/// The campaign number in the start of a Patreon feed.
fn patreon_campaign(head: &[u8]) -> Option<String> {
let text = String::from_utf8_lossy(head);
text.match_indices("patreon.com/rss/").find_map(|(i, m)| {
let id: String = text[i + m.len()..].chars().take_while(char::is_ascii_digit).collect();
(!id.is_empty()).then_some(id)
})
}
/// A campaign's name and its shows as (id, title), from Patreon's JSON:API answer.
fn parse_patreon_shows(json: &[u8]) -> Result<(Option<String>, Vec<(String, String)>)> {
let v: serde_json::Value = serde_json::from_slice(json).context("Patreon's answer is not JSON")?;
// Missing is not the same as none. Read as no shows, the creator feed would be scanned as
// a plain feed, claim every show's files, and leave the shows empty once the list returned.
let ids = v["data"]["relationships"]["shows"]["data"]
.as_array()
.ok_or_else(|| anyhow!("Patreon's answer does not list the shows"))?;
let title = |id: &str| -> Option<String> {
let show = v["included"].as_array()?.iter().find(|x| x["type"] == "collection" && x["id"] == id)?;
show["attributes"]["title"].as_str().map(|t| t.trim().to_owned())
};
let shows = ids
.iter()
.filter_map(|s| s["id"].as_str())
.map(|id| (id.to_owned(), title(id).unwrap_or_else(|| format!("Show {id}"))))
.collect();
Ok((v["data"]["attributes"]["name"].as_str().map(str::to_owned), shows))
}
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`. /// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> { pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
match rss::Channel::read_from(bytes) { match rss::Channel::read_from(bytes) {
@@ -550,6 +647,45 @@ mod tests {
assert!(!is_opml(include_bytes!("../tests/data/atom.xml"))); assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
} }
#[test]
fn a_patreon_creator_is_a_list_of_its_shows() {
let tok = "AbCdEfGhIjKlMnOpQrStUvWxYz012_-9";
assert_eq!(expand_input(&format!(" {tok} ")), format!("https://www.patreon.com/rss?auth={tok}"));
assert_eq!(expand_input("https://example.com/rss"), "https://example.com/rss");
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss/glasscannon?auth={tok}")));
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss?auth={tok}")));
assert!(!is_patreon_creator(&format!("https://www.patreon.com/rss/x?auth={tok}&show=1")), "one show is a feed");
assert!(!is_patreon_creator(&format!("https://example.com/rss?auth={tok}")));
// The show you already have by name is the one a bare token would add by number.
assert!(same_feed(
&format!("https://www.patreon.com/rss/glasscannon?auth={tok}&show=2073588"),
&format!("https://www.patreon.com/rss?auth={tok}&show=2073588"),
));
assert!(!same_feed(
&format!("https://www.patreon.com/rss?auth={tok}&show=1"),
&format!("https://www.patreon.com/rss?auth={tok}&show=2"),
));
// The self link carries the campaign by number, whichever spelling was asked for.
let head = br#"<rss><channel><link>https://www.patreon.com/glasscannon</link>
<atom:link href="https://www.patreon.com/rss/369921?auth=t" rel="self"/>"#;
assert_eq!(patreon_campaign(head).as_deref(), Some("369921"));
assert_eq!(patreon_campaign(b"<rss><channel><title>T"), None);
let json = br#"{"data":{"id":"369921","type":"campaign","attributes":{"name":"The Glass Cannon Network"},
"relationships":{"shows":{"data":[{"id":"2073588","type":"collection"},{"id":"2073636","type":"collection"}]}}},
"included":[{"id":"2073588","type":"collection","attributes":{"title":"Get in the Trunk "}},
{"id":"2073636","type":"collection","attributes":{"title":"Shadowdark"}}]}"#;
let (name, shows) = parse_patreon_shows(json).unwrap();
assert_eq!(name.as_deref(), Some("The Glass Cannon Network"));
assert_eq!(shows, [("2073588".into(), "Get in the Trunk".into()), ("2073636".into(), "Shadowdark".into())]);
// An answer that stops naming the shows is an error, never "this creator has none".
assert!(parse_patreon_shows(br#"{"data":{"attributes":{"name":"X"}}}"#).is_err());
}
#[test] #[test]
fn an_item_may_carry_several_enclosures() { fn an_item_may_carry_several_enclosures() {
// The rss crate keeps only one per item -- the last -- so these come from the XML. // The rss crate keeps only one per item -- the last -- so these come from the XML.

View File

@@ -111,18 +111,11 @@ impl Visit for Collect {
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) { fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
self.add(field, format!("{value:?}")); self.add(field, format!("{value:?}"));
} }
// Numbers and bools reach record_debug through the trait's defaults, which prints them the
// same way. A string would print quoted there, hence its own method.
fn record_str(&mut self, field: &Field, value: &str) { fn record_str(&mut self, field: &Field, value: &str) {
self.add(field, value.to_owned()); self.add(field, value.to_owned());
} }
fn record_i64(&mut self, field: &Field, value: i64) {
self.add(field, value.to_string());
}
fn record_u64(&mut self, field: &Field, value: u64) {
self.add(field, value.to_string());
}
fn record_bool(&mut self, field: &Field, value: bool) {
self.add(field, value.to_string());
}
} }
#[cfg(test)] #[cfg(test)]

View File

@@ -334,8 +334,7 @@ async fn daemon(
anyhow::bail!("a daemon is already listening on {}", socket.display()); anyhow::bail!("a daemon is already listening on {}", socket.display());
} }
// A database with nobody in it cannot be signed into, and an install that predates // A database with nobody in it cannot be signed into.
// accounts still has to serve its owner. Both get the same starting point.
if ctx.db.users()?.is_empty() { if ctx.db.users()?.is_empty() {
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?; ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
tracing::warn!( tracing::warn!(
@@ -344,22 +343,15 @@ async fn daemon(
); );
} }
// A library that predates accounts belongs to whoever was using it: the admin.
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) { if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect(); let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
match ctx.db.adopt_existing_library(admin.id, &catalogue) { match ctx.db.adopt_catalogue(admin.id, &catalogue) {
Ok(0) => {} Ok(0) => {}
Ok(n) => tracing::info!(user = %admin.name, entries = n, "adopted the existing library"), Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
Err(e) => tracing::error!(error = %e, "could not adopt the existing library"), Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
} }
} }
match migrate_opml_children(&ctx) {
Ok(n) if n > 0 => tracing::info!(count = n, "moved OPML feeds out of config.toml into the database"),
Ok(_) => {}
Err(e) => tracing::warn!(error = ?e, "could not tidy OPML feeds out of the config"),
}
match ctx.db.requeue_interrupted() { match ctx.db.requeue_interrupted() {
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"), Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
Ok(_) => {} Ok(_) => {}
@@ -468,7 +460,7 @@ async fn start_web(
let mut fresh = (*cfg).clone(); let mut fresh = (*cfg).clone();
fresh.web.enabled = true; fresh.web.enabled = true;
fresh.web.bind = bind.clone(); fresh.web.bind = bind.clone();
fresh.web.token = web::generate_token(); fresh.web.token = crate::auth::new_session_token();
fresh.save(config_path)?; fresh.save(config_path)?;
ctx.reload_cfg(config_path)?; ctx.reload_cfg(config_path)?;
println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token); println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token);
@@ -517,8 +509,9 @@ async fn add(
keywords: Vec<String>, keywords: Vec<String>,
) -> Result<()> { ) -> Result<()> {
let mut cfg = (*ctx.cfg()).clone(); let mut cfg = (*ctx.cfg()).clone();
let url = &feed::expand_input(url);
// Includes feeds derived from an OPML, or the same show could be added twice. // Includes feeds derived from an OPML, or the same show could be added twice.
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| s.cfg.url == url) { if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
anyhow::bail!("already subscribed as {:?}", existing.id); anyhow::bail!("already subscribed as {:?}", existing.id);
} }
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?; let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
@@ -569,10 +562,17 @@ pub async fn add_one(
// Slugs must be unique across derived feeds too, or a new feed can collide with one // Slugs must be unique across derived feeds too, or a new feed can collide with one
// an OPML already introduced. // an OPML already introduced.
let taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)? let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
.into_iter() .into_iter()
.map(|s| (s.id, s.cfg)) .map(|s| (s.id, s.cfg))
.collect(); .collect();
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
// it gets its history back, and a different feed does not inherit someone else's.
for (id, other) in ctx.db.feed_urls()? {
if !feed::same_feed(&other, url) {
taken.entry(id).or_insert_with(|| probe.clone());
}
}
let id = config::unique_slug(&title, &taken); let id = config::unique_slug(&title, &taken);
cfg.feeds.insert(id.clone(), probe); cfg.feeds.insert(id.clone(), probe);
Ok(id) Ok(id)
@@ -807,7 +807,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
ctx.out.emit(Event::FeedSkip { ctx.out.emit(Event::FeedSkip {
feed: id.clone(), feed: id.clone(),
reason: format!( reason: format!(
"OPML: {total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing", "{total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
added.len() added.len()
), ),
}); });
@@ -908,36 +908,6 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
Ok(out) Ok(out)
} }
/// Moves OPML children that older versions wrote into config.toml over to the database.
/// They were never yours to edit, and 80-odd of them made the file unreadable.
fn migrate_opml_children(ctx: &Ctx) -> Result<usize> {
let cfg = (*ctx.cfg()).clone();
let children: Vec<(String, config::Feed)> = cfg
.feeds
.iter()
.filter(|(_, f)| f.group.is_some())
.map(|(id, f)| (id.clone(), f.clone()))
.collect();
if children.is_empty() {
return Ok(0);
}
let mut fresh = cfg.clone();
for (id, f) in &children {
let group = f.group.clone().unwrap_or_default();
let title = ctx
.db
.feed_summary(id)
.ok()
.and_then(|s| s.title)
.unwrap_or_else(|| id.clone());
ctx.db.upsert_managed(id, &f.url, &title, &group)?;
fresh.feeds.remove(id);
}
fresh.save(&ctx.config_path)?;
ctx.reload_cfg(&ctx.config_path)?;
Ok(children.len())
}
/// Seconds to wait before re-checking a feed. /// Seconds to wait before re-checking a feed.
/// ///
/// A per-feed schedule is an explicit instruction and wins outright. Without one, the /// A per-feed schedule is an explicit instruction and wins outright. Without one, the
@@ -963,7 +933,7 @@ struct Scan {
enum Outcome { enum Outcome {
NotModified, NotModified,
Feed(Scan), Feed(Scan),
/// The URL served an OPML document, so it is a subscription list rather than a feed. /// The URL is a list of feeds rather than a feed: an OPML, or a Patreon creator's shows.
Opml { added: Vec<String>, removed: usize, kept: usize, total: usize }, Opml { added: Vec<String>, removed: usize, kept: usize, total: usize },
} }
@@ -973,6 +943,31 @@ async fn scan_one(
feed_cfg: &config::Feed, feed_cfg: &config::Feed,
state: &db::HttpState, state: &db::HttpState,
) -> Result<Outcome> { ) -> Result<Outcome> {
// A Patreon creator with more than one show is a list of feeds, like an OPML.
if feed::is_patreon_creator(&feed_cfg.url) {
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
Ok((name, shows)) if shows.len() > 1 => {
ctx.db.touch_feed(id, &feed_cfg.url)?;
if let Some(name) = name {
ctx.db.set_title(id, &name)?;
}
// Read as one feed before it was split, it listed every show's items in one
// heap. The items go; its files and read state move to each show as the show
// lists them (`Db::adopt`), so no show comes up empty for want of a URL.
ctx.db.clear_entries(id)?;
return sync_group(ctx, id, feed_cfg, &shows).await;
}
Ok(_) => {} // One show: the creator's feed is that show.
// Already split: keep the shows it has rather than read the creator as one heap.
Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e),
Err(e) => tracing::warn!(
feed = id,
error = %format!("{e:#}"),
"could not list the Patreon shows; reading it as one feed"
),
}
}
let mut fetched = feed::fetch( let mut fetched = feed::fetch(
&ctx.client, &ctx.client,
feed_cfg, feed_cfg,
@@ -1018,19 +1013,38 @@ async fn scan_one(
)?; )?;
let policy = policy_for(ctx, id, feed_cfg)?; let policy = policy_for(ctx, id, feed_cfg)?;
if let Some(parent) = &feed_cfg.group {
let listed: Vec<(&str, &str)> = parsed
.entries
.iter()
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
.collect();
ctx.db.adopt(parent, id, &listed)?;
}
// Verdicts are recorded in `state`, so the download queue below is just "everything still
// pending". A filter's verdict is looked at again on every scan, though: made once, at
// discovery, it outlived the setting behind it, and allowing explicit items afterwards
// changed nothing however often the feed was scanned.
let skipped = ctx.db.skipped_by_filter(id)?;
let mut scan = Scan::default(); let mut scan = Scan::default();
for entry in &parsed.entries { for entry in &parsed.entries {
if ctx.db.record_entry(id, entry)? { if ctx.db.record_entry(id, entry)? {
scan.new_entries += 1; scan.new_entries += 1;
} }
for enc in &entry.enclosures { for enc in &entry.enclosures {
if !ctx.db.record_enclosure(id, &entry.guid, enc)? { let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? {
continue; // Seen before: downloaded, skipped or deliberately reaped. None
} else if let Some(reason) = skipped.get(&enc.url) {
Some(reason.as_str())
} else {
continue; // Settled: queued, downloaded, reaped, or another feed's file.
};
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
if now != was {
match now {
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?,
None => ctx.db.mark_enclosure(&enc.url, "pending", None)?,
} }
// Filters run once, at discovery, and are recorded in `state`. The download
// queue below is then just "everything still pending".
if let Some(reason) = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc) {
ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?;
} }
} }
} }
@@ -1114,12 +1128,6 @@ async fn scan_one(
Ok(Outcome::Feed(scan)) Ok(Outcome::Feed(scan))
} }
/// Brings the feed list in step with a subscribed OPML.
///
/// New entries are added under the OPML's group and folder. An entry that has gone from
/// the OPML is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
/// the UI to explain them.
async fn sync_opml( async fn sync_opml(
ctx: &Arc<Ctx>, ctx: &Arc<Ctx>,
parent_id: &str, parent_id: &str,
@@ -1130,25 +1138,44 @@ async fn sync_opml(
if let Some(title) = feed::opml_title(bytes) { if let Some(title) = feed::opml_title(bytes) {
ctx.db.set_title(parent_id, &title)?; ctx.db.set_title(parent_id, &title)?;
} }
sync_group(ctx, parent_id, parent, &listed).await
}
/// Brings the feed list in step with a list of feeds: a subscribed OPML, or a Patreon
/// creator's shows.
///
/// New entries are added under the list's group and folder. An entry that has gone from
/// the list is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
/// the UI to explain them.
async fn sync_group(
ctx: &Arc<Ctx>,
parent_id: &str,
parent: &config::Feed,
listed: &[(String, String)],
) -> Result<Outcome> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
let existing = ctx.db.managed_feeds()?; let existing = ctx.db.managed_feeds()?;
let mut added = vec![]; let mut added = vec![];
for (title, url) in &listed { for (title, url) in listed {
// Already known, whether derived or promoted into the config. // Already known, whether derived or promoted into the config.
if let Some(m) = existing.iter().find(|m| &m.url == url) { if let Some(m) = existing.iter().find(|m| &m.url == url) {
ctx.db.upsert_managed(&m.id, url, title, parent_id)?; ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
continue; continue;
} }
if cfg.feeds.values().any(|f| &f.url == url) { // A Patreon show you added by hand may be spelled differently from the one listed.
if cfg.feeds.values().any(|f| feed::same_feed(&f.url, url)) {
continue; continue;
} }
// A removed feed keeps its rows, so its id is only free again for the same feed.
let known = ctx.db.feed_urls()?;
let taken: std::collections::BTreeMap<String, config::Feed> = cfg let taken: std::collections::BTreeMap<String, config::Feed> = cfg
.feeds .feeds
.keys() .keys()
.chain(existing.iter().map(|m| &m.id)) .chain(existing.iter().map(|m| &m.id))
.chain(added.iter()) .chain(added.iter())
.chain(known.iter().filter(|(_, u)| !feed::same_feed(u, url)).map(|(id, _)| id))
.map(|id| (id.clone(), parent.clone())) .map(|id| (id.clone(), parent.clone()))
.collect(); .collect();
let id = config::unique_slug(title, &taken); let id = config::unique_slug(title, &taken);
@@ -1255,7 +1282,7 @@ pub struct Policy {
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> { fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
let global = ctx.cfg().general.max_new_per_check; let global = ctx.cfg().general.max_new_per_check;
Ok(merge_policy(&ctx.db.subscribers(id)?, feed_cfg, global)) Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global))
} }
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy { fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {

View File

@@ -12,9 +12,7 @@ use axum::{
}, },
routing::{delete, get, patch, post}, routing::{delete, get, patch, post},
}; };
use futures_util::StreamExt;
use serde::Deserialize; use serde::Deserialize;
use tokio_stream::wrappers::BroadcastStream;
use tower::ServiceExt; use tower::ServiceExt;
use tower_http::services::ServeFile; use tower_http::services::ServeFile;
use serde::Serialize; use serde::Serialize;
@@ -35,28 +33,6 @@ pub struct WebState {
pub events: broadcast::Sender<Event>, pub events: broadcast::Sender<Event>,
} }
/// A 32-hex-character shared secret, generated when config.toml has none.
///
/// ponytail: /dev/urandom rather than a CSPRNG crate -- 16 bytes, once, on a Unix-only
/// binary. Falls back to the clock only if urandom is somehow unreadable, which would be a
/// weak token, so that case is logged loudly.
pub fn generate_token() -> String {
use std::io::Read;
let mut bytes = [0u8; 16];
match std::fs::File::open("/dev/urandom").and_then(|mut f| f.read_exact(&mut bytes)) {
Ok(()) => {}
Err(e) => {
tracing::error!(error = %e, "could not read /dev/urandom; token is NOT secure");
let n = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as u64)
.unwrap_or(0);
bytes[..8].copy_from_slice(&n.to_le_bytes());
}
}
bytes.iter().map(|b| format!("{b:02x}")).collect()
}
pub fn router(state: WebState) -> Router { pub fn router(state: WebState) -> Router {
Router::new() Router::new()
.route("/", get(index)) .route("/", get(index))
@@ -88,6 +64,7 @@ pub fn router(state: WebState) -> Router {
// Signing in cannot require being signed in, so these sit outside the auth layer. // Signing in cannot require being signed in, so these sit outside the auth layer.
.route("/login", get(login_page)) .route("/login", get(login_page))
.route("/api/login", post(login)) .route("/api/login", post(login))
.route("/icon.png", get(icon))
.layer(middleware::from_fn(access_log)) .layer(middleware::from_fn(access_log))
.with_state(state) .with_state(state)
} }
@@ -433,6 +410,15 @@ async fn login_page() -> Html<&'static str> {
Html(include_str!("../web/login.html")) Html(include_str!("../web/login.html"))
} }
/// The 2004 icon, served once for both pages rather than inlined as base64 into each. The
/// sign-in page shows it, so it sits outside the auth layer with /login.
async fn icon() -> impl IntoResponse {
(
[(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")],
include_bytes!("../web/ipodderx-icon.png").as_slice(),
)
}
fn constant_time_eq(a: &str, b: &str) -> bool { fn constant_time_eq(a: &str, b: &str) -> bool {
let (a, b) = (a.as_bytes(), b.as_bytes()); let (a, b) = (a.as_bytes(), b.as_bytes());
if a.len() != b.len() { if a.len() != b.len() {
@@ -495,6 +481,9 @@ async fn feeds(
let mut out = Vec::with_capacity(mine.len()); let mut out = Vec::with_capacity(mine.len());
for sub in &subs { for sub in &subs {
let (id, feed) = (&sub.id, &sub.cfg); let (id, feed) = (&sub.id, &sub.cfg);
// In a group, what you have not set on the feed comes from your settings on the group,
// the same fallback the scanner uses (`Db::subscribers`).
let up = feed.group.as_deref().and_then(|g| mine.get(g));
let Some(mine) = mine.get(id) else { continue }; let Some(mine) = mine.get(id) else { continue };
let s = state.ctx.db.feed_summary(id)?; let s = state.ctx.db.feed_summary(id)?;
let st = state.ctx.db.http_state(id)?; let st = state.ctx.db.http_state(id)?;
@@ -504,11 +493,22 @@ async fn feeds(
title: s.title, title: s.title,
image: s.image, image: s.image,
folder: feed.folder.clone(), folder: feed.folder.clone(),
keywords: mine.keywords.clone().unwrap_or_else(|| feed.keywords.clone()), keywords: mine
allow_explicit: mine.allow_explicit.unwrap_or(feed.allow_explicit), .keywords
auto_download: mine.auto_download.unwrap_or(feed.auto_download), .clone()
.or_else(|| up.and_then(|u| u.keywords.clone()))
.unwrap_or_else(|| feed.keywords.clone()),
allow_explicit: mine
.allow_explicit
.or(up.and_then(|u| u.allow_explicit))
.unwrap_or(feed.allow_explicit),
auto_download: mine
.auto_download
.or(up.and_then(|u| u.auto_download))
.unwrap_or(feed.auto_download),
max_new_per_check: mine max_new_per_check: mine
.max_new_per_check .max_new_per_check
.or(up.and_then(|u| u.max_new_per_check))
.map(|n| n as usize) .map(|n| n as usize)
.or(feed.max_new_per_check), .or(feed.max_new_per_check),
group: feed.group.clone(), group: feed.group.clone(),
@@ -803,15 +803,6 @@ mod tests {
"another feed already has that URL" "another feed already has that URL"
); );
} }
#[test]
fn generated_tokens_are_32_hex_chars_and_not_repeated() {
let a = generate_token();
let b = generate_token();
assert_eq!(a.len(), 32);
assert!(a.chars().all(|c| c.is_ascii_hexdigit()));
assert_ne!(a, b);
}
} }
#[derive(Deserialize)] #[derive(Deserialize)]
@@ -893,6 +884,19 @@ struct NewFeed {
folder: Option<String>, folder: Option<String>,
#[serde(default)] #[serde(default)]
keywords: Vec<String>, keywords: Vec<String>,
#[serde(default)]
allow_explicit: bool,
}
/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it
/// goes on your subscription, and before the first scan, which would otherwise skip every
/// explicit item.
fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
if allow {
let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() };
state.ctx.db.set_subscription(user_id, &sub)?;
}
Ok(())
} }
async fn add_feed( async fn add_feed(
@@ -901,23 +905,28 @@ async fn add_feed(
Json(body): Json<NewFeed>, Json(body): Json<NewFeed>,
) -> Result<Json<serde_json::Value>, ApiError> { ) -> Result<Json<serde_json::Value>, ApiError> {
let mut cfg = (*state.ctx.cfg()).clone(); let mut cfg = (*state.ctx.cfg()).clone();
let url = crate::feed::expand_input(&body.url);
// Someone else may already have it. Then adding costs nothing: no second fetch, no // Someone else may already have it. Then adding costs nothing: no second fetch, no
// second copy on disk, just another name against the same feed. // second copy on disk, just another name against the same feed.
if let Some(existing) = crate::subscriptions(&state.ctx)? if let Some(existing) = crate::subscriptions(&state.ctx)?
.into_iter() .into_iter()
.find(|s| s.cfg.url == body.url) .find(|s| crate::feed::same_feed(&s.cfg.url, &url))
{ {
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some(); let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some();
state.ctx.db.subscribe(user.id, &existing.id)?; state.ctx.db.subscribe(user.id, &existing.id)?;
if !already {
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit)?;
}
scan_soon(&state, Some(existing.id.clone())).await; scan_soon(&state, Some(existing.id.clone())).await;
return Ok(Json( return Ok(Json(
serde_json::json!({ "id": existing.id, "existing": already }), serde_json::json!({ "id": existing.id, "existing": already }),
)); ));
} }
let id = crate::add_one(&state.ctx, &mut cfg, &body.url, body.folder, body.keywords).await?; let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
cfg.save(&state.config_path)?; cfg.save(&state.config_path)?;
state.ctx.reload_cfg(&state.config_path)?; state.ctx.reload_cfg(&state.config_path)?;
state.ctx.db.subscribe(user.id, &id)?; state.ctx.db.subscribe(user.id, &id)?;
explicit_on_add(&state, user.id, &id, body.allow_explicit)?;
scan_soon(&state, Some(id.clone())).await; scan_soon(&state, Some(id.clone())).await;
Ok(Json(serde_json::json!({ "id": id, "existing": false }))) Ok(Json(serde_json::json!({ "id": id, "existing": false })))
} }
@@ -1164,9 +1173,9 @@ async fn delete_file(
let complaint = match (starred, unread) { let complaint = match (starred, unread) {
(0, 0) => None, (0, 0) => None,
(0, u) => Some(format!("{} subscribed to this feed {} not played it yet", people(u), if u == 1 { "has" } else { "have" })), (0, u) => Some(format!("{} subscribed to this feed {} not played it yet", people(u), if u == 1 { "has" } else { "have" })),
(st, 0) => Some(format!("another {} starred it to keep", people(st))), (st, 0) => Some(format!("another {} kept it", people(st))),
(st, u) => Some(format!( (st, u) => Some(format!(
"another {} starred it to keep, and {} not played it yet", "another {} kept it, and {} not played it yet",
people(st), people(st),
if u == 1 { "one person has".to_string() } else { format!("{u} have") } if u == 1 { "one person has".to_string() } else { format!("{u} have") }
)), )),
@@ -1215,9 +1224,20 @@ async fn fetch_now(
/// The same broadcast the socket clients read, as server-sent events. /// The same broadcast the socket clients read, as server-sent events.
async fn events(State(state): State<WebState>) -> Sse<impl futures_util::Stream<Item = Result<SseEvent, std::convert::Infallible>>> { async fn events(State(state): State<WebState>) -> Sse<impl futures_util::Stream<Item = Result<SseEvent, std::convert::Infallible>>> {
let stream = BroadcastStream::new(state.events.subscribe()).filter_map(|ev| async move { // A client that falls behind skips what it missed rather than being cut off.
let ev = ev.ok()?; let stream = futures_util::stream::unfold(state.events.subscribe(), |mut rx| async move {
Some(Ok(SseEvent::default().data(serde_json::to_string(&ev).ok()?))) loop {
match rx.recv().await {
Ok(ev) => {
if let Ok(data) = serde_json::to_string(&ev) {
let ev = Ok::<_, std::convert::Infallible>(SseEvent::default().data(data));
return Some((ev, rx));
}
}
Err(broadcast::error::RecvError::Lagged(_)) => {}
Err(broadcast::error::RecvError::Closed) => return None,
}
}
}); });
Sse::new(stream).keep_alive(axum::response::sse::KeepAlive::default()) Sse::new(stream).keep_alive(axum::response::sse::KeepAlive::default())
} }
@@ -1428,8 +1448,6 @@ async fn patch_settings(
))); )));
} }
cfg.general.schedule = sched; cfg.general.schedule = sched;
// The legacy key would otherwise keep shadowing intent in the file.
cfg.general.interval_mins = None;
} }
if let Some(v) = body.max_new_per_check { if let Some(v) = body.max_new_per_check {
cfg.general.max_new_per_check = v; cfg.general.max_new_per_check = v;

View File

@@ -142,7 +142,7 @@ test('the filter tabs change what is listed', async ({ page }) => {
await page.locator('.tabs button', { hasText: 'Unread' }).first().click(); await page.locator('.tabs button', { hasText: 'Unread' }).first().click();
expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all); expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all);
await page.locator('.tabs button', { hasText: 'Flagged' }).first().click(); await page.locator('.tabs button', { hasText: 'Kept' }).first().click();
await expect(page.locator('#count')).toContainText('0 items'); await expect(page.locator('#count')).toContainText('0 items');
}); });
@@ -187,9 +187,11 @@ test('an OPML subscription is a collapsible folder', async ({ page }) => {
body: JSON.stringify({ feed: 'test-subscriptions', force: true }), body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
})); }));
// Every row reserves the chevron slot for alignment; only a folder's is clickable. // Only a folder has a triangle, and it is a button that says whether the folder is open.
const chev = page.locator('.feed.group .chev'); const chev = page.locator('.feed.group .chev');
await expect(chev).toBeVisible({ timeout: 20_000 }); await expect(chev).toBeVisible({ timeout: 20_000 });
await expect(page.locator('.feed:not(.group) .chev')).toHaveCount(0);
await expect(chev).toHaveAttribute('aria-expanded', 'false');
// Closed by default: the children are not listed until the folder is opened. // Closed by default: the children are not listed until the folder is opened.
const before = await page.locator('.feed').count(); const before = await page.locator('.feed').count();
@@ -346,6 +348,10 @@ test('a second person has their own feeds and their own read state', async ({ br
const ctx = await browser.newContext(); const ctx = await browser.newContext();
const page = await ctx.newPage(); const page = await ctx.newPage();
await page.goto('/login'); await page.goto('/login');
// The sign-in page shows the icon, so it has to load before anyone has signed in.
const icon = await page.request.get('/icon.png');
expect(icon.status()).toBe(200);
expect(icon.headers()['content-type']).toBe('image/png');
await page.locator('#name').fill('sam'); await page.locator('#name').fill('sam');
await page.locator('#pw').fill('sampassword'); await page.locator('#pw').fill('sampassword');
await page.locator('button[type=submit]').click(); await page.locator('button[type=submit]').click();

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long