Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9eb7aadced | |||
| c6bceaef37 | |||
| dc63d6acaf | |||
| 8937f35f00 | |||
| 0990f2a90d | |||
| 0cc002cdfa | |||
| 2c9e899762 | |||
| 2e416f96cf | |||
| 9269aa99f7 |
62
CHANGELOG.md
62
CHANGELOG.md
@@ -10,6 +10,65 @@ The long form, with what was wrong before and how it was found, is in
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.5.0] - 2026-09-12
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- A Patreon token pasted into Add feed, or a creator's RSS link without `&show=`, becomes a folder
|
||||||
|
of that creator's shows, kept in step on every scan like a subscribed OPML. A creator with only
|
||||||
|
one show stays a plain feed. One already added as a single long feed is split into its shows on
|
||||||
|
its next scan, keeping its files and what you had read.
|
||||||
|
- Add feed has an "Allow items marked explicit" box, so a new feed's first scan no longer skips
|
||||||
|
every explicit item.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Unread counts, unread dots and download progress are amber, the colour of the icon's EQ bars.
|
||||||
|
Blue is kept for the primary action and links, so a count no longer looks like a button.
|
||||||
|
- What is playing is marked by small EQ bars, in its row and in the player. They move only while it
|
||||||
|
plays.
|
||||||
|
- A folder in the sidebar shows its first four shows' art as a mosaic, and its shows sit under its
|
||||||
|
title. Only folders have a triangle, so every feed lines up with Directory and Popular above.
|
||||||
|
- Feeds without art get initials in a colour of their own, instead of all the same grey.
|
||||||
|
- The Flagged tab is called Kept, as the Keep button and Settings already said.
|
||||||
|
- A feed's header is one short line; when it checks next is in its tooltip.
|
||||||
|
- The item list takes more of the window, and the Files pane shows only when the item has files.
|
||||||
|
- Column headings and tags are in sentence case, and fewer things are bold.
|
||||||
|
- Unsubscribe is a round button beside the feed's other actions.
|
||||||
|
- Export and Import in Settings say what they do.
|
||||||
|
- The sign-in page shows the original icon large.
|
||||||
|
- Nothing animates when your system asks for reduced motion.
|
||||||
|
- The pages are about 90 KB smaller: the icon is served once instead of written into each.
|
||||||
|
- A web token generated for a new install is 64 characters instead of 32.
|
||||||
|
- The README is a short overview of what ipx does and how to run it, and points into `docs/` for
|
||||||
|
the rest. It still described the layout from before 0.4.0.
|
||||||
|
|
||||||
|
### Removed
|
||||||
|
|
||||||
|
- The systemd units in `contrib/`. Run ipx with Docker, or point a unit of your own at
|
||||||
|
`ipx daemon`.
|
||||||
|
- Upgrading from before 0.3.0 directly: what was read, kept or part-played before accounts is no
|
||||||
|
longer carried over to the admin, and OPML feeds that old versions wrote into `config.toml` are
|
||||||
|
no longer moved out of it. Upgrade through 0.4.0 first.
|
||||||
|
- `interval_mins` in `config.toml` is ignored; use `schedule`.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- The feed list works from the keyboard: Tab reaches every feed, Enter opens it, and Right and Left
|
||||||
|
open and close a folder. Every button shows where the focus is, including in the toolbar, which
|
||||||
|
used to clip the ring.
|
||||||
|
- "1 items" reads "1 item".
|
||||||
|
- A selected feed without art no longer loses its initials tile in Dark and Light.
|
||||||
|
- The player shows the feed's initials when there is no art, not the episode's.
|
||||||
|
- Turning on Allow explicit, or changing keywords or auto-download, brings back what those settings
|
||||||
|
had skipped on the feed's next scan. Before, an item was judged once, when first seen, and a
|
||||||
|
skipped one stayed skipped whatever you changed.
|
||||||
|
- Feeds inside an OPML or a Patreon creator follow your settings on the folder unless you set their
|
||||||
|
own, as the folder's settings dialog said they did. Before, the folder's settings reached nothing
|
||||||
|
inside it.
|
||||||
|
- A new feed no longer takes the name of one you removed earlier and shows that feed's old items.
|
||||||
|
Re-adding the same feed still gets its old name, and its history, back.
|
||||||
|
|
||||||
## [0.4.0] - 2026-09-11
|
## [0.4.0] - 2026-09-11
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
@@ -199,7 +258,8 @@ The long form, with what was wrong before and how it was found, is in
|
|||||||
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
|
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
|
||||||
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
|
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
|
||||||
|
|
||||||
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...main
|
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.0...main
|
||||||
|
[0.5.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...v0.5.0
|
||||||
[0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0
|
[0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0
|
||||||
[0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0
|
[0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0
|
||||||
[0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0
|
[0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0
|
||||||
|
|||||||
10
CLAUDE.md
10
CLAUDE.md
@@ -99,9 +99,9 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
|
|||||||
|
|
||||||
* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
|
* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
|
||||||
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
|
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
|
||||||
* **`entries.read`, `entries.flagged` and `entries.position` are dead columns.** Read state lives in
|
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
|
||||||
`entry_state` per user. Two bugs have already come from queries still reading the old ones
|
and `position` columns from before accounts; two bugs came from queries still reading them
|
||||||
(retention, and the entry pruner) — grep before adding a third.
|
(retention, and the entry pruner), and `migrate()` now drops them.
|
||||||
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
|
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
|
||||||
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
|
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
|
||||||
derived and never written to config.
|
derived and never written to config.
|
||||||
@@ -115,6 +115,10 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
|
|||||||
fighting over the database, with the stale one still holding the port.
|
fighting over the database, with the stale one still holding the port.
|
||||||
* `/api/settings` answering `200` does **not** mean the worker is alive — it is a different task.
|
* `/api/settings` answering `200` does **not** mean the worker is alive — it is a different task.
|
||||||
Probe the control socket (`ipx status`) to check that.
|
Probe the control socket (`ipx status`) to check that.
|
||||||
|
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's
|
||||||
|
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on
|
||||||
|
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait
|
||||||
|
for `daemon started` in the log. Copy `state.db` aside before deploying one.
|
||||||
|
|
||||||
## House style
|
## House style
|
||||||
|
|
||||||
|
|||||||
34
Cargo.lock
generated
34
Cargo.lock
generated
@@ -436,17 +436,6 @@ dependencies = [
|
|||||||
"shlex",
|
"shlex",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "cfb"
|
|
||||||
version = "0.14.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "a347dcabdae9c31b0825fd6a8bed285ec9c2acb89c47827126d52fa4f59cece3"
|
|
||||||
dependencies = [
|
|
||||||
"fnv",
|
|
||||||
"uuid",
|
|
||||||
"web-time",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "cfg-if"
|
name = "cfg-if"
|
||||||
version = "1.0.4"
|
version = "1.0.4"
|
||||||
@@ -867,15 +856,6 @@ dependencies = [
|
|||||||
"dirs-sys",
|
"dirs-sys",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "dirs"
|
|
||||||
version = "7.0.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "8d57d423b3c82e89b9a24ca3091fee61f456a26edbd28d26c65906f4bc1dcd8f"
|
|
||||||
dependencies = [
|
|
||||||
"dirs-sys",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "dirs-sys"
|
name = "dirs-sys"
|
||||||
version = "0.5.0"
|
version = "0.5.0"
|
||||||
@@ -1608,15 +1588,6 @@ dependencies = [
|
|||||||
"serde_core",
|
"serde_core",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "infer"
|
|
||||||
version = "0.22.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "f4200d433cbd5178df7797c9c2e75b348b728e39631cf14520d1e2fc424201f4"
|
|
||||||
dependencies = [
|
|
||||||
"cfb",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "intervaltree"
|
name = "intervaltree"
|
||||||
version = "0.2.7"
|
version = "0.2.7"
|
||||||
@@ -1634,7 +1605,7 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "ipx"
|
name = "ipx"
|
||||||
version = "0.4.0"
|
version = "0.5.0"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"ammonia",
|
"ammonia",
|
||||||
"anyhow",
|
"anyhow",
|
||||||
@@ -1643,9 +1614,7 @@ dependencies = [
|
|||||||
"axum",
|
"axum",
|
||||||
"chrono",
|
"chrono",
|
||||||
"clap",
|
"clap",
|
||||||
"dirs",
|
|
||||||
"futures-util",
|
"futures-util",
|
||||||
"infer",
|
|
||||||
"librqbit",
|
"librqbit",
|
||||||
"opml",
|
"opml",
|
||||||
"percent-encoding",
|
"percent-encoding",
|
||||||
@@ -1656,7 +1625,6 @@ dependencies = [
|
|||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
"tokio",
|
"tokio",
|
||||||
"tokio-stream",
|
|
||||||
"toml",
|
"toml",
|
||||||
"tower",
|
"tower",
|
||||||
"tower-http 0.7.1",
|
"tower-http 0.7.1",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "ipx"
|
name = "ipx"
|
||||||
version = "0.4.0"
|
version = "0.5.0"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
@@ -11,9 +11,7 @@ atom_syndication = "0.12.10"
|
|||||||
axum = "0.8.9"
|
axum = "0.8.9"
|
||||||
chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] }
|
chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] }
|
||||||
clap = { version = "4.6.6", features = ["derive"] }
|
clap = { version = "4.6.6", features = ["derive"] }
|
||||||
dirs = "7.0.0"
|
|
||||||
futures-util = { version = "0.3.34", default-features = false, features = ["std"] }
|
futures-util = { version = "0.3.34", default-features = false, features = ["std"] }
|
||||||
infer = "0.22.0"
|
|
||||||
librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] }
|
librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] }
|
||||||
opml = "1.1.6"
|
opml = "1.1.6"
|
||||||
percent-encoding = "2.3.2"
|
percent-encoding = "2.3.2"
|
||||||
@@ -24,7 +22,6 @@ rusqlite = { version = "0.40.2", features = ["bundled"] }
|
|||||||
serde = { version = "1.0.229", features = ["derive"] }
|
serde = { version = "1.0.229", features = ["derive"] }
|
||||||
serde_json = "1.0.151"
|
serde_json = "1.0.151"
|
||||||
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
|
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
|
||||||
tokio-stream = { version = "0.1.19", features = ["sync"] }
|
|
||||||
toml = "1.1.5"
|
toml = "1.1.5"
|
||||||
tower = { version = "0.5.3", features = ["util"] }
|
tower = { version = "0.5.3", features = ["util"] }
|
||||||
tower-http = { version = "0.7.1", features = ["fs"] }
|
tower-http = { version = "0.7.1", features = ["fs"] }
|
||||||
|
|||||||
144
README.md
144
README.md
@@ -1,123 +1,83 @@
|
|||||||
# ipodderx-rs
|
# ipodderx-rs
|
||||||
|
|
||||||
A headless podcatcher: scans RSS/Atom feeds, downloads enclosures (HTTP and BitTorrent), files them
|
A self-hosted podcatcher for a household. It checks your feeds, downloads the episodes, and serves
|
||||||
into per-feed folders, and reaps old files to stay under a disk quota. Runs as a one-shot CLI or as
|
a web UI modelled on the 2004 Mac app **iPodderX**, for any number of people sharing one copy of
|
||||||
a daemon with a web UI, serving any number of people from one copy of the data.
|
the files. One Rust binary, `ipx`, is both the daemon and the command line.
|
||||||
|
|
||||||
## Lineage
|
It is a rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python engine
|
||||||
|
behind iPodderX (2004-2008, Ray Slakinski & August Trometer).
|
||||||
|
|
||||||
A modern Rust rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python 2
|
## What it does
|
||||||
engine behind **iPodderX** (2004-2008, Ray Slakinski & August Trometer), open-sourced under the MIT
|
|
||||||
License in 2010.
|
|
||||||
|
|
||||||
What carries over: the feed scan and TTL handling, GUID/URL dedupe, per-feed and per-date download
|
- **The web UI.** It has a toolbar, and a feed list that opens with Directory, Popular and All
|
||||||
folders, keyword filters, the explicit-content filter, torrent enclosures, and "SmartSpace" -- the
|
Subscriptions. Items sit in a sortable table with a Files pane, and there is a player bar. It
|
||||||
oldest-first disk quota reaper.
|
comes in Dark, Light and Classic themes, and works on a phone.
|
||||||
|
- **Several people, one copy.** Each person has their own subscriptions and their own read, kept
|
||||||
|
and playback state. There is one file on disk per episode, however many people want it. People
|
||||||
|
sign in with a password or through a proxy (Cloudflare Zero Trust or Authentik), and admins
|
||||||
|
manage accounts and settings.
|
||||||
|
- **Scanning.** Feeds are checked on a schedule, globally or per feed, and a feed's own TTL is
|
||||||
|
honoured. Keyword, explicit-content and media-type filters decide what is downloaded, with a cap
|
||||||
|
on new downloads per scan.
|
||||||
|
- **Downloads.** Files come over HTTP or BitTorrent and are filed into a folder per feed.
|
||||||
|
Retention deletes the oldest files to stay under a disk quota or an age limit, and never touches
|
||||||
|
an item someone has kept.
|
||||||
|
- **OPML.** You can import and export your own subscriptions. You can also subscribe to an OPML
|
||||||
|
URL, which keeps a whole list in step as a folder.
|
||||||
|
|
||||||
What does not: iTunes and iPhoto export via AppleScript, text-to-speech enclosures, the Windows
|
## Run it
|
||||||
WMP/COM paths, XML plists and Python pickles for state, the `directory.iPodderX.com` survey ping,
|
|
||||||
3DES-encrypted preferences, and the `printMSG` stdout protocol -- replaced by a JSON-lines socket.
|
|
||||||
|
|
||||||
## Quick start
|
With Docker:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker build -t ipodderx .
|
||||||
|
docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
`docker-compose.yml` is set up for the author's own server. Point its `image` and its three volumes
|
||||||
|
(`/config`, `/data` and `/downloads`) at yours first. The UI is on port 8099. BitTorrent uses 6881
|
||||||
|
over TCP and UDP. Files are written as `PUID`/`PGID`, 99:100 by default.
|
||||||
|
|
||||||
|
From source:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cargo build --release
|
cargo build --release
|
||||||
install -m755 target/release/ipx ~/.cargo/bin/
|
./target/release/ipx daemon
|
||||||
|
|
||||||
ipx add https://atp.fm/rss # subscribe
|
|
||||||
ipx fetch # scan and download
|
|
||||||
ipx daemon # scheduler, control socket and web UI
|
|
||||||
```
|
```
|
||||||
|
|
||||||
On first start with `[web] enabled = true` the daemon mints a token, writes it to config.toml and
|
The first start creates **admin / ipodderx**. Sign in at `/login`, then change it:
|
||||||
prints the URL to open. A database with no accounts starts with **admin / ipodderx** at `/login` --
|
|
||||||
change it with `echo -n '<password>' | ipx user passwd admin`.
|
```sh
|
||||||
|
echo -n 'a good password' | ipx user passwd admin
|
||||||
|
```
|
||||||
|
|
||||||
|
The UI is plain HTTP, so put TLS in front of it if it is reachable from outside your network.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
| [docs/configuration.md](docs/configuration.md) | Every config key, paths, environment variables |
|
| [docs/configuration.md](docs/configuration.md) | Every config key, path and environment variable |
|
||||||
| [docs/cli.md](docs/cli.md) | Every command, including `ipx user` |
|
| [docs/cli.md](docs/cli.md) | Every command, including `ipx user` |
|
||||||
| [docs/users.md](docs/users.md) | Accounts, and what several people share |
|
| [docs/users.md](docs/users.md) | Accounts, and what several people share |
|
||||||
| [docs/sso.md](docs/sso.md) | Cloudflare Zero Trust or Authentik in front |
|
| [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik |
|
||||||
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, socket, HTTP API |
|
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API |
|
||||||
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
|
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
|
||||||
| [docs/history.md](docs/history.md) | How it was built: the long form, with what was wrong and why |
|
| [docs/history.md](docs/history.md) | How it was built, with what was wrong and why |
|
||||||
| [CLAUDE.md](CLAUDE.md) | Notes for anyone (or anything) working on the code |
|
| [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed |
|
||||||
|
|
||||||
## The web UI
|
|
||||||
|
|
||||||
`ipx daemon` serves it in the same process, so it reads SQLite and the event bus directly.
|
|
||||||
|
|
||||||
Feeds down the side; the selected feed's items across the top; the selected item's text and its
|
|
||||||
enclosures below, which is where you play, download or delete them. The divider drags and its
|
|
||||||
position is remembered. Playback serves Range requests, so seeking works. An OPML subscription is a
|
|
||||||
collapsible folder whose page lists the feeds inside it.
|
|
||||||
|
|
||||||
An item may carry several enclosures; all of them appear below, and anything that is not audio or
|
|
||||||
video gets a View link rather than a player -- the publisher's copy until it is downloaded, the
|
|
||||||
local one after. Opening an item marks it read. Show notes are untrusted feed HTML, sanitized with
|
|
||||||
`ammonia` server-side before they reach the page.
|
|
||||||
|
|
||||||
The **Log** button shows the running daemon live in four tabs: *Daemon I/O* is the control protocol
|
|
||||||
itself, every command in and event out; *Scans* is feed and download activity; *HTTP* is web
|
|
||||||
requests; *All* is everything, with level and text filters and a copy button. It reads a ring buffer
|
|
||||||
held in the process, not a file, so it works the same under Docker.
|
|
||||||
|
|
||||||
It is plain HTTP. On a LAN bind the token and everything else cross the network in the clear, and a
|
|
||||||
feed URL can itself carry a credential. Put TLS in front of it if that matters.
|
|
||||||
|
|
||||||
## OPML
|
|
||||||
|
|
||||||
**Importing and exporting** a file copies subscriptions in or out once: `ipx import subs.opml`,
|
|
||||||
`ipx export subs.opml`, or Settings → Subscriptions in the UI.
|
|
||||||
|
|
||||||
**Subscribing to an OPML URL** is a live subscription, as iPodderX had. Add the OPML's URL like any
|
|
||||||
other feed; every scan re-reads it and keeps your list in step. The feeds inside are not written to
|
|
||||||
config.toml -- the OPML is the source of truth, so they are re-derived each scan and held in the
|
|
||||||
database. They show as a folder, download into one nested folder, and inherit the subscription's
|
|
||||||
settings until you change one, which gives it its own entry.
|
|
||||||
|
|
||||||
When a feed drops out of the OPML upstream, it is unsubscribed and removed -- unless it has
|
|
||||||
downloads, in which case it is kept and flagged in the UI as no longer listed. A downloaded file is
|
|
||||||
never left behind with nothing explaining where it came from.
|
|
||||||
|
|
||||||
## Docker
|
|
||||||
|
|
||||||
```sh
|
|
||||||
docker buildx build --tag 192.168.1.130:5000/ipodderx:latest . --push
|
|
||||||
docker compose pull ipodderx && docker compose up -d ipodderx
|
|
||||||
docker compose logs -f ipodderx # the first start prints the default admin password
|
|
||||||
```
|
|
||||||
|
|
||||||
`docker-compose.yml` runs the image from the registry above rather than building it, so build and
|
|
||||||
push first; change the tag in both places to use another registry. It mounts `/config` (config.toml),
|
|
||||||
`/data` (state.db) and `/downloads` from this install's host paths, which you will want to change for
|
|
||||||
yours. It publishes 8099 for the
|
|
||||||
UI and 6881 (TCP **and** UDP -- DHT needs the UDP side), and sets `PUID`/`PGID` to `99:100` so files
|
|
||||||
land owned the way Unraid shares expect. The healthcheck runs `ipx status` through the control
|
|
||||||
socket, so it catches a daemon that is alive but wedged rather than merely one that has died.
|
|
||||||
|
|
||||||
## Running it as a service
|
|
||||||
|
|
||||||
`contrib/` has a systemd user unit for the daemon, and a timer plus one-shot service if you would
|
|
||||||
rather run periodic scans with no daemon -- in which case there is no socket for a UI to attach to.
|
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state
|
cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state
|
||||||
node tests/page-smoke.js # the page script loads without throwing
|
node tests/page-smoke.js # the page script loads without throwing
|
||||||
npx playwright test # a real browser against a real daemon
|
npx playwright test # a real browser against a real daemon on fixture feeds
|
||||||
```
|
```
|
||||||
|
|
||||||
`npm install` gets the test runner; the browser comes from
|
`npm install` gets the test runner, and `npx playwright install --with-deps chromium` gets the
|
||||||
`npx playwright install --with-deps chromium` (in `install.sh`).
|
browser.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
MIT. See [LICENSE](LICENSE).
|
MIT, see [LICENSE](LICENSE). The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by
|
||||||
|
@fontawesome, under [CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG.
|
||||||
The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by @fontawesome, under
|
|
||||||
[CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG in `web/index.html`.
|
|
||||||
|
|||||||
30
TODO.md
Normal file
30
TODO.md
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
# To do
|
||||||
|
|
||||||
|
## Cut what is no longer needed
|
||||||
|
|
||||||
|
From a whole-repo audit for over-engineering on 2026-09-12. Biggest cut first.
|
||||||
|
|
||||||
|
- [x] **Pre-accounts adoption and the dead `entries` columns.** The copy of the old read state into
|
||||||
|
`entry_state` and the `entries.read`, `flagged` and `position` columns are gone. Its other half,
|
||||||
|
subscribing the first admin to the catalogue, was not dead and stays as `adopt_catalogue`.
|
||||||
|
(`src/db.rs`, `src/main.rs`)
|
||||||
|
- [x] **`contrib/` systemd units.** From before the container; nothing points at them.
|
||||||
|
- [x] **`migrate_opml_children`.** A one-time move of OPML children out of `config.toml` that has
|
||||||
|
run. Delete it and its call. (`src/main.rs`)
|
||||||
|
- [x] **The legacy `interval_mins` key.** Production uses `schedule`. Delete the field, the fallback
|
||||||
|
in `General::interval` and its test. (`src/config.rs`)
|
||||||
|
- [x] **`Db::entries` and `Db::count_entries`.** One-line wrappers only the tests call; the tests
|
||||||
|
call `entries_in` and `count_in` instead. (`src/db.rs`)
|
||||||
|
- [x] **`web::generate_token`.** Repeats `auth::new_session_token`. Use that. (`src/web.rs`)
|
||||||
|
- [x] **Page leftovers.** `globalEvery`, `S.busy`, `S.limit`, `unitOptions`' `firstLabel`, `--r`,
|
||||||
|
`.ep.open`, the phone `.ep .art`, the duplicate phone `.fhead.slim{flex-wrap}`, the second
|
||||||
|
`#sidebar{z-index}`, and the `on()` helper. (`web/index.html`)
|
||||||
|
- [x] **`logbuf` visitors.** `record_i64`, `record_u64` and `record_bool` repeat what `Visit`'s
|
||||||
|
defaults already do through `record_debug`. (`src/logbuf.rs`)
|
||||||
|
- [x] **The `infer` dependency.** Its torrent check is the `d8:announce` test on the next line.
|
||||||
|
- [x] **The `dirs` dependency.** `XDG_CONFIG_HOME`, `XDG_DATA_HOME` and `HOME` from `std::env`.
|
||||||
|
- [x] **The `tokio-stream` dependency.** `futures_util::stream::unfold` over the broadcast receiver.
|
||||||
|
- [x] **The icon inlined four times.** About 94 KB of base64 across both pages; serve it once as
|
||||||
|
`/icon.png` from `include_bytes!`, open without signing in like `/login`.
|
||||||
|
|
||||||
|
After these: `cargo test`, `node tests/page-smoke.js`, `npx playwright test`.
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=ipx feed scan (one shot)
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=oneshot
|
|
||||||
ExecStart=%h/.cargo/bin/ipx fetch
|
|
||||||
Environment=IPX_LOG=ipx=info
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
# User unit: install to ~/.config/systemd/user/ipx.service, then
|
|
||||||
# systemctl --user enable --now ipx
|
|
||||||
# The socket lands in $XDG_RUNTIME_DIR/ipx.sock by default, so a UI running as the
|
|
||||||
# same user can attach without extra configuration.
|
|
||||||
[Unit]
|
|
||||||
Description=ipx podcatcher
|
|
||||||
After=network-online.target
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=simple
|
|
||||||
ExecStart=%h/.cargo/bin/ipx daemon
|
|
||||||
Restart=on-failure
|
|
||||||
RestartSec=30
|
|
||||||
Environment=IPX_LOG=ipx=info
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=default.target
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
# Alternative to the daemon: a periodic one-shot scan, closer to how the original
|
|
||||||
# iPodderX agent was driven. Use this OR ipx.service, not both -- with no daemon
|
|
||||||
# running there is no socket, so a UI cannot attach.
|
|
||||||
#
|
|
||||||
# Install ipx-scan.service and ipx.timer to ~/.config/systemd/user/, then
|
|
||||||
# systemctl --user enable --now ipx.timer
|
|
||||||
[Unit]
|
|
||||||
Description=Periodic ipx feed scan
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
OnBootSec=5min
|
|
||||||
OnUnitActiveSec=1h
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
@@ -28,12 +28,15 @@ The page is compiled in, so **editing `web/index.html` needs a rebuild**.
|
|||||||
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
|
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
|
||||||
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
|
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
|
||||||
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
|
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
|
||||||
re-derived into the database each scan, never written to config.toml.
|
re-derived into the database each scan, never written to config.toml. A Patreon creator link
|
||||||
|
(a token, no `show=`) with more than one show is treated the same way, before any fetch: its
|
||||||
|
shows come from Patreon's web API and each becomes a derived feed.
|
||||||
4. Record entries. A changed title or description flips the item back to unread.
|
4. Record entries. A changed title or description flips the item back to unread.
|
||||||
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
|
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
|
||||||
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
|
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
|
||||||
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
|
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
|
||||||
a reason.
|
a reason. What a filter skipped is judged again every scan, so a change of settings brings it
|
||||||
|
back. A feed in a group takes your settings on the group for anything you have not set on it.
|
||||||
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
|
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
|
||||||
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
|
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
|
||||||
or an error page — and is deleted.
|
or an error page — and is deleted.
|
||||||
@@ -55,9 +58,9 @@ entry_state user_id, feed_id, guid, read, flagged, position
|
|||||||
PK (user_id, feed_id, guid)
|
PK (user_id, feed_id, guid)
|
||||||
```
|
```
|
||||||
|
|
||||||
`entries` still has `read`, `flagged` and `position` columns from before accounts existed. They are
|
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
|
||||||
**dead** — the migration copied them into `entry_state` and nothing reads them now. Anything found
|
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
|
||||||
querying them is a bug; two were.
|
older database.
|
||||||
|
|
||||||
Schema changes: add the table or column to `SCHEMA`, and for a column also to the list in
|
Schema changes: add the table or column to `SCHEMA`, and for a column also to the list in
|
||||||
`migrate()`, which does `PRAGMA table_info` then `ALTER TABLE ADD COLUMN`. `Db::memory()` runs the
|
`migrate()`, which does `PRAGMA table_info` then `ALTER TABLE ADD COLUMN`. `Db::memory()` runs the
|
||||||
@@ -108,7 +111,7 @@ else a `401`.
|
|||||||
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
|
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
|
||||||
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
|
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
|
||||||
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
|
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
|
||||||
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, starred, position |
|
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, kept, position |
|
||||||
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
|
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
|
||||||
| `POST /api/fetch` | |
|
| `POST /api/fetch` | |
|
||||||
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
|
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
|
||||||
|
|||||||
@@ -60,7 +60,7 @@ ipx reap # actually delete
|
|||||||
```
|
```
|
||||||
|
|
||||||
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
|
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
|
||||||
with no file are pruned from the database. **Starred by anyone keeps a file**, and one only counts
|
with no file are pruned from the database. **An item anyone kept keeps its file**, and one only counts
|
||||||
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
|
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
|
||||||
stops the next scan fetching it again.
|
stops the next scan fetching it again.
|
||||||
|
|
||||||
|
|||||||
@@ -32,10 +32,10 @@ media_types = ["audio", "video"]
|
|||||||
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
|
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
|
||||||
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
|
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
|
||||||
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
|
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
|
||||||
pad. Starred items are never deleted, and a file only counts as read once every subscriber has
|
pad. Kept items are never deleted, and a file only counts as read once every subscriber has
|
||||||
read it. `0` disables it entirely.
|
read it. `0` disables it entirely.
|
||||||
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
|
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
|
||||||
Starred ones stay. `0` disables it.
|
Kept ones stay. `0` disables it.
|
||||||
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
|
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
|
||||||
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
|
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
|
||||||
fetched 216 files and 22 GB in one scan.
|
fetched 216 files and 22 GB in one scan.
|
||||||
@@ -43,8 +43,6 @@ media_types = ["audio", "video"]
|
|||||||
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
|
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
|
||||||
without this the disk fills with artwork. Empty takes everything.
|
without this the disk fills with artwork. Empty takes everything.
|
||||||
|
|
||||||
`interval_mins` from older configs is still read, and `schedule` supersedes it.
|
|
||||||
|
|
||||||
## `[torrent]`
|
## `[torrent]`
|
||||||
|
|
||||||
```toml
|
```toml
|
||||||
|
|||||||
130
docs/history.md
130
docs/history.md
@@ -6,6 +6,136 @@ reasoning lives. New write-ups go at the top.
|
|||||||
|
|
||||||
See [README.md](../README.md) for what the thing is.
|
See [README.md](../README.md) for what the thing is.
|
||||||
|
|
||||||
|
## 2026-09-12 — Cutting what had outlived its reason
|
||||||
|
|
||||||
|
A whole-repo audit for over-engineering listed twelve things to cut, and all of them went.
|
||||||
|
|
||||||
|
- **Upgrades from before accounts.** `migrate_opml_children` moved OPML feeds that old versions
|
||||||
|
wrote into `config.toml` out to the database, and ran at every daemon start to do nothing after
|
||||||
|
the first. Production ran it in 0.3.0; anything older has to pass through 0.4.0.
|
||||||
|
- **Half of the adoption, and not the other half.** The audit called `adopt_existing_library` a
|
||||||
|
one-time migration and it was cut whole. It did two jobs: copy the old read state into
|
||||||
|
`entry_state`, which was dead, and subscribe the first admin to the whole catalogue while nobody
|
||||||
|
subscribed to anything, which is how a fresh install's first account gets `config.toml`'s feeds.
|
||||||
|
The browser suite caught it at once, signing in to an empty sidebar; `cargo test` had no idea.
|
||||||
|
The second job is back as `adopt_catalogue`, with a unit test of its own.
|
||||||
|
- **The dead `entries` columns.** `read`, `flagged` and `position` moved to `entry_state` with
|
||||||
|
accounts. The adoption's copy was their last reader, but `record_entry` still wrote them, and
|
||||||
|
still reset `read` when a title changed, which nothing looked at. Two bugs came from queries
|
||||||
|
reading them. `migrate()` now drops them from an existing database (SQLite has had `DROP COLUMN`
|
||||||
|
since 3.35), and a test builds an old table to prove it. On production each drop rewrote the
|
||||||
|
66 MB `entries` table, about four seconds apiece, so the first start took thirteen. An
|
||||||
|
`ipx status` run in that window failed with `migrating schema`: every `ipx` command migrates when
|
||||||
|
it opens the database, and it collided with the daemon doing the same. A failed `ALTER TABLE`
|
||||||
|
changes nothing, and the database had been copied to `backup/` first anyway.
|
||||||
|
- **`interval_mins`**, which `schedule` replaced. An old config that still has the key loads; the
|
||||||
|
key is ignored, and the config test carries it to keep that true.
|
||||||
|
- **Three dependencies.** `infer` was only asked whether a file is a torrent, and the check after
|
||||||
|
it already looked for `d8:announce`, which is what `infer` looks for. `dirs` was three lookups of
|
||||||
|
`XDG_CONFIG_HOME`, `XDG_DATA_HOME` and `HOME`. `tokio-stream` wrapped the broadcast receiver for
|
||||||
|
the event stream; `futures_util::stream::unfold` does the same, lagging clients included.
|
||||||
|
- **Two token generators.** The web token came from a copy of the session-token code, with a
|
||||||
|
clock fallback on top. It uses `auth::new_session_token` now, and is 64 characters.
|
||||||
|
- **The icon inlined four times**, 23 KB of base64 each, into both pages. It is `/icon.png` now,
|
||||||
|
outside the sign-in wall with `/login`, since the sign-in page shows it.
|
||||||
|
- Also: the `contrib/` systemd units from before the container, `Db::entries` and
|
||||||
|
`Db::count_entries` that only the tests called, three `logbuf` visitors that repeated the trait's
|
||||||
|
defaults, and unused state, a helper and dead CSS in the page.
|
||||||
|
|
||||||
|
## 2026-09-11 — A design pass on the web UI
|
||||||
|
|
||||||
|
A review against screenshots of every view in all three themes found that Dark and Light read as a
|
||||||
|
generic dark dashboard: one pale blue did every job, most labels were bold, and nothing led. The
|
||||||
|
list it produced, in `TODO.md`, was worked through in one go. What was worth knowing:
|
||||||
|
|
||||||
|
- **Amber means new.** Badges, unread dots and download bars take the icon's EQ amber; blue is left
|
||||||
|
for the primary action and links. Light's amber was `#b06f10`, which gives white text 4.1:1,
|
||||||
|
short of AA for 11 px bold. It is `#9a5f0a` now, 5.2:1.
|
||||||
|
- **EQ bars mark what is playing.** Three `<i>` bars stand at 60, 100 and 40 % and animate only
|
||||||
|
while `body.playing` is set. The first version left the animation on but paused, expecting each
|
||||||
|
bar to hold a different frame. The frames it held were within a pixel of each other, and on
|
||||||
|
screen the bars read as three dots. Under reduced motion one rule drops every animation and
|
||||||
|
transition, which leaves the bars standing.
|
||||||
|
- **The focus ring was clipped.** `.tgroup` and `#topbar` both set `overflow:hidden`, so a ring
|
||||||
|
drawn outside a toolbar button was cut off. Rings inside clipping parents are inset instead.
|
||||||
|
- **The feed list could not be used from the keyboard at all.** Rows were `<div>`s and the triangle
|
||||||
|
a `<span>`, so Tab went from the feed filter to Sign out. Rows now take focus, the triangle is a
|
||||||
|
`<button aria-expanded>`, and `renderFeeds` puts focus back on the same row after redrawing,
|
||||||
|
since every live update replaces every row. The list's own key handler stops Space and the
|
||||||
|
arrows from reaching the player's shortcuts on the document.
|
||||||
|
- **The triangle hangs in the margin.** Every row used to reserve an 18 px slot for it, pushing a
|
||||||
|
hundred feeds 28 px right of the places above for the sake of two folders. It is now absolutely
|
||||||
|
placed in the row's left padding, the full height of the row, so a near miss no longer opens
|
||||||
|
the folder's page.
|
||||||
|
- **A selected tile vanished** because the initials tile and the selected row were both `--raise`.
|
||||||
|
Tiles now mix their tint into `--bg`, which no row uses.
|
||||||
|
- **The Files pane hides itself** with `#split:has(>#files[hidden])`, which collapses its column.
|
||||||
|
The phone layout already hides the pane, and a zero-width extra track there is harmless.
|
||||||
|
- **Flagged became Kept** in the tab, and in the server's refusal to delete a file someone else
|
||||||
|
kept. The filter value and the column stay `flagged`; renaming those buys nothing.
|
||||||
|
|
||||||
|
## 2026-09-11 — A Patreon creator is a list of shows
|
||||||
|
|
||||||
|
Ray asked whether ipx could sync with Patreon. Not in full. The documented API (v2, the
|
||||||
|
`identity.memberships` scope) lists the creators you back and whether each has a feed (`has_rss`),
|
||||||
|
but no resource carries the `auth` token that makes a feed URL work. That token only comes from the
|
||||||
|
creator's page. It is also one per membership, not one per account: techpod's differs from Glass
|
||||||
|
Cannon's, so no single token finds everything you back.
|
||||||
|
|
||||||
|
What does work is one creator at a time, which is what Ray wanted for Glass Cannon and its 33 shows:
|
||||||
|
|
||||||
|
- `patreon.com/rss?auth=<token>`, with no creator named, returns that token's creator. Its self link,
|
||||||
|
about 660 bytes in, gives the campaign by number (`/rss/369921`). Patreon ignores `Range` here, so
|
||||||
|
ipx reads the stream until the number appears and hangs up, instead of taking all 2.8 MB.
|
||||||
|
- A show's `show=` number is a Patreon collection. Asked anonymously, the collection listing
|
||||||
|
(`/api/collection?filter[campaign_id]=`) and a post's `collections` both hide patron-only ones: you
|
||||||
|
get "FAQ". `/api/campaigns/<id>?include=shows` lists every show, anonymously, in one response.
|
||||||
|
- Every spelling works: `rss/glasscannon?auth=…&show=N`, `rss/369921?…` and `rss?auth=…&show=N` all
|
||||||
|
serve the same 131 items. Enclosure URLs are the same in the creator feed and the show feed, and
|
||||||
|
stable between fetches.
|
||||||
|
|
||||||
|
That last point shaped the design. `enclosures.url` is unique, so whichever feed is scanned first owns
|
||||||
|
the file. The first cut only asked a creator for its shows while it had no entries of its own, so that a
|
||||||
|
creator already read as a plain feed, holding every show's episodes, would never be split into shows
|
||||||
|
that came up empty. Within the hour that was the wrong call: Glass Cannon had gone into production on
|
||||||
|
the build before this one, been read as one feed of 2,385 items, and the rule kept it that way. Finding
|
||||||
|
anything in that heap was the problem Ray wanted solved.
|
||||||
|
|
||||||
|
So a creator with more than one show is always a group, run through the same sync as an OPML
|
||||||
|
(`sync_group`, split out of `sync_opml`). When it becomes one, its items are cleared and each show
|
||||||
|
takes over the enclosures the creator holds as the show lists them (`Db::adopt`), downloaded files and
|
||||||
|
everyone's read state included. One show leaves it a plain feed, which is what techpod already was. If
|
||||||
|
the shows cannot be listed, a creator already split fails the scan rather than being read as one heap;
|
||||||
|
one that never was is read as one feed until they can be. An answer without a `shows` list is an error,
|
||||||
|
not "no shows".
|
||||||
|
|
||||||
|
**Filter verdicts follow the settings.** Ray also reported that turning on Allow explicit and
|
||||||
|
rescanning brought nothing back. An item was judged once, when first seen, and `skipped` was final. The
|
||||||
|
2026-09-10 entry below saw it coming ("worth a `ipx retry <feed>` command if this bites"). It bit:
|
||||||
|
2,166 Glass Cannon items and all 88 of Shadowdark's sat at `skipped: explicit` with the setting on.
|
||||||
|
Every scan now runs the filters again over what they skipped (not over `torrents disabled`, which is not
|
||||||
|
a filter's call) and requeues what they now let through. Only that direction: a queued item is never
|
||||||
|
pulled back, because Download latest and a manual download both work by queueing.
|
||||||
|
|
||||||
|
**Two gaps beside it.** Add feed had no explicit box, so every new feed's first scan skipped all its
|
||||||
|
explicit items; it has one now, stored on your subscription like the feed dialog's. And a feed inside a
|
||||||
|
group ignored your settings on the group, though the group's dialog said they were inherited: settings
|
||||||
|
live on each person's subscription, and nothing read the group's. `Db::subscribers` now fills what you
|
||||||
|
have not set on the feed from your subscription to the group, and the feed list shows the same.
|
||||||
|
|
||||||
|
**A name that was already used.** Replaying the split on a copy of the production database left one
|
||||||
|
show with a Supercast episode in it. "Glass Cannon Live! Ascension | Pathfinder 2E" slugs to
|
||||||
|
`glass-cannon-live-ascension-pathfinder-2`, the id of a Supercast feed of the same show that had been
|
||||||
|
removed. Removing a feed keeps its rows on purpose, so that re-adding it does not fetch the back
|
||||||
|
catalogue again, but choosing a new id only checked config.toml and derived feeds. The Patreon show took
|
||||||
|
the old id and everything still filed under it. An id is now also taken when the database has a feed by
|
||||||
|
that id at a different URL; the same URL may still have it back, which is the re-add case.
|
||||||
|
|
||||||
|
Shows already added by hand are matched by token and show number, not by exact URL (`same_feed`), so a
|
||||||
|
bare token does not add Get in the Trunk and Shadowdark a second time under another spelling.
|
||||||
|
|
||||||
|
The show listing is Patreon's own undocumented web API. If it changes, only finding new shows stops.
|
||||||
|
|
||||||
## 2026-09-11 — One meaning per icon, sortable columns, and one player
|
## 2026-09-11 — One meaning per icon, sortable columns, and one player
|
||||||
|
|
||||||
Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act
|
Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ fetch, one parse and one file.
|
|||||||
|
|
||||||
| Yours alone | The same for everyone |
|
| Yours alone | The same for everyone |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Read, starred, playback position | The feed's URL |
|
| Read, kept, playback position | The feed's URL |
|
||||||
| Which feeds you see at all | Its download folder |
|
| Which feeds you see at all | Its download folder |
|
||||||
| Keywords, auto-download, explicit, per-scan cap | When it is scanned |
|
| Keywords, auto-download, explicit, per-scan cap | When it is scanned |
|
||||||
| | The file on disk |
|
| | The file on disk |
|
||||||
@@ -34,10 +34,10 @@ is shared.
|
|||||||
|
|
||||||
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
|
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
|
||||||
for everyone** and names them in the confirmation, and the server has the last word: if anyone else
|
for everyone** and names them in the confirmation, and the server has the last word: if anyone else
|
||||||
has starred the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
|
has kept the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
|
||||||
reason, and only `?force=true` goes through.
|
reason, and only `?force=true` goes through.
|
||||||
|
|
||||||
Retention follows the same rule: starred by anyone keeps a file, and it counts as read only once
|
Retention follows the same rule: an item anyone kept keeps its file, and it counts as read only once
|
||||||
every subscriber has read it.
|
every subscriber has read it.
|
||||||
|
|
||||||
## Signing in
|
## Signing in
|
||||||
|
|||||||
@@ -26,9 +26,6 @@ pub struct General {
|
|||||||
/// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d".
|
/// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d".
|
||||||
/// A feed's own `schedule` overrides this.
|
/// A feed's own `schedule` overrides this.
|
||||||
pub schedule: String,
|
pub schedule: String,
|
||||||
/// Superseded by `schedule`. Still read so existing configs keep working.
|
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
|
||||||
pub interval_mins: Option<u64>,
|
|
||||||
pub organize: Organize,
|
pub organize: Organize,
|
||||||
/// 0 = unlimited.
|
/// 0 = unlimited.
|
||||||
pub max_total_gb: f64,
|
pub max_total_gb: f64,
|
||||||
@@ -153,7 +150,6 @@ impl Default for General {
|
|||||||
download_dir: home().join("Podcasts"),
|
download_dir: home().join("Podcasts"),
|
||||||
socket: default_socket(),
|
socket: default_socket(),
|
||||||
schedule: "every 60m".into(),
|
schedule: "every 60m".into(),
|
||||||
interval_mins: None,
|
|
||||||
organize: Organize::Feed,
|
organize: Organize::Feed,
|
||||||
max_total_gb: 0.0,
|
max_total_gb: 0.0,
|
||||||
max_age_days: 0,
|
max_age_days: 0,
|
||||||
@@ -176,8 +172,8 @@ impl Default for Torrent {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl General {
|
impl General {
|
||||||
/// Minutes between checks. Falls back to the legacy `interval_mins`, then to an hour.
|
/// Minutes between checks, or an hour when `schedule` is empty or unreadable. A malformed
|
||||||
/// A malformed value warns rather than stopping the daemon.
|
/// value warns rather than stopping the daemon.
|
||||||
pub fn interval(&self) -> u64 {
|
pub fn interval(&self) -> u64 {
|
||||||
if let Some(n) = parse_interval(&self.schedule) {
|
if let Some(n) = parse_interval(&self.schedule) {
|
||||||
return n;
|
return n;
|
||||||
@@ -185,7 +181,7 @@ impl General {
|
|||||||
if !self.schedule.trim().is_empty() {
|
if !self.schedule.trim().is_empty() {
|
||||||
tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default");
|
tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default");
|
||||||
}
|
}
|
||||||
self.interval_mins.filter(|n| *n > 0).unwrap_or(60)
|
60
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -278,9 +274,7 @@ pub fn config_path() -> PathBuf {
|
|||||||
if let Ok(p) = std::env::var("IPX_CONFIG") {
|
if let Ok(p) = std::env::var("IPX_CONFIG") {
|
||||||
return PathBuf::from(p);
|
return PathBuf::from(p);
|
||||||
}
|
}
|
||||||
dirs::config_dir()
|
xdg("XDG_CONFIG_HOME", ".config").join("ipx/config.toml")
|
||||||
.unwrap_or_else(|| home().join(".config"))
|
|
||||||
.join("ipx/config.toml")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`.
|
/// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`.
|
||||||
@@ -288,9 +282,7 @@ pub fn data_dir() -> PathBuf {
|
|||||||
if let Ok(p) = std::env::var("IPX_DATA_DIR") {
|
if let Ok(p) = std::env::var("IPX_DATA_DIR") {
|
||||||
return PathBuf::from(p);
|
return PathBuf::from(p);
|
||||||
}
|
}
|
||||||
dirs::data_dir()
|
xdg("XDG_DATA_HOME", ".local/share").join("ipx")
|
||||||
.unwrap_or_else(|| home().join(".local/share"))
|
|
||||||
.join("ipx")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn default_socket() -> PathBuf {
|
fn default_socket() -> PathBuf {
|
||||||
@@ -346,8 +338,16 @@ pub fn unique_slug(text: &str, taken: &BTreeMap<String, Feed>) -> String {
|
|||||||
(2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap()
|
(2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `$var`, or `~/fallback` when it is unset or empty, as the XDG base directory spec says.
|
||||||
|
fn xdg(var: &str, fallback: &str) -> PathBuf {
|
||||||
|
std::env::var_os(var)
|
||||||
|
.filter(|v| !v.is_empty())
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|| home().join(fallback))
|
||||||
|
}
|
||||||
|
|
||||||
fn home() -> PathBuf {
|
fn home() -> PathBuf {
|
||||||
dirs::home_dir().unwrap_or_else(|| PathBuf::from("."))
|
std::env::var_os("HOME").map(PathBuf::from).unwrap_or_else(|| PathBuf::from("."))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn expand_tilde(p: &Path) -> PathBuf {
|
fn expand_tilde(p: &Path) -> PathBuf {
|
||||||
@@ -367,6 +367,8 @@ mod tests {
|
|||||||
r#"
|
r#"
|
||||||
[general]
|
[general]
|
||||||
download_dir = "/tmp/pods"
|
download_dir = "/tmp/pods"
|
||||||
|
# A key older versions read. An old config that still has it has to load.
|
||||||
|
interval_mins = 45
|
||||||
|
|
||||||
[feeds.example]
|
[feeds.example]
|
||||||
url = "https://example.com/feed.xml"
|
url = "https://example.com/feed.xml"
|
||||||
@@ -403,22 +405,17 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn interval_falls_back_through_legacy_then_default() {
|
fn interval_falls_back_to_an_hour() {
|
||||||
let mut g = General::default();
|
let mut g = General::default();
|
||||||
assert_eq!(g.interval(), 60, "the default schedule");
|
assert_eq!(g.interval(), 60, "the default schedule");
|
||||||
|
|
||||||
g.schedule = "every 15m".into();
|
g.schedule = "every 15m".into();
|
||||||
assert_eq!(g.interval(), 15);
|
assert_eq!(g.interval(), 15);
|
||||||
|
|
||||||
// A config written before `schedule` existed still works.
|
// Empty or garbage must not stop the daemon.
|
||||||
g.schedule = String::new();
|
g.schedule = String::new();
|
||||||
g.interval_mins = Some(45);
|
assert_eq!(g.interval(), 60);
|
||||||
assert_eq!(g.interval(), 45);
|
|
||||||
|
|
||||||
// Garbage must not stop the daemon.
|
|
||||||
g.schedule = "whenever".into();
|
g.schedule = "whenever".into();
|
||||||
assert_eq!(g.interval(), 45);
|
|
||||||
g.interval_mins = None;
|
|
||||||
assert_eq!(g.interval(), 60);
|
assert_eq!(g.interval(), 60);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
292
src/db.rs
292
src/db.rs
@@ -40,14 +40,10 @@ CREATE TABLE IF NOT EXISTS entries (
|
|||||||
published INTEGER,
|
published INTEGER,
|
||||||
description TEXT,
|
description TEXT,
|
||||||
first_seen INTEGER NOT NULL,
|
first_seen INTEGER NOT NULL,
|
||||||
read INTEGER NOT NULL DEFAULT 0,
|
|
||||||
flagged INTEGER NOT NULL DEFAULT 0,
|
|
||||||
image TEXT,
|
image TEXT,
|
||||||
duration INTEGER,
|
duration INTEGER,
|
||||||
episode INTEGER,
|
episode INTEGER,
|
||||||
season INTEGER,
|
season INTEGER,
|
||||||
-- Seconds into the audio, so playback resumes where it was left.
|
|
||||||
position INTEGER NOT NULL DEFAULT 0,
|
|
||||||
PRIMARY KEY (feed_id, guid)
|
PRIMARY KEY (feed_id, guid)
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -93,7 +89,7 @@ CREATE TABLE IF NOT EXISTS subscriptions (
|
|||||||
PRIMARY KEY (user_id, feed_id)
|
PRIMARY KEY (user_id, feed_id)
|
||||||
);
|
);
|
||||||
|
|
||||||
-- Read, starred and how far in. One row per person per item, created on first touch;
|
-- Read, kept and how far in. One row per person per item, created on first touch;
|
||||||
-- an item nobody has touched has no row at all, which is what unread means.
|
-- an item nobody has touched has no row at all, which is what unread means.
|
||||||
CREATE TABLE IF NOT EXISTS entry_state (
|
CREATE TABLE IF NOT EXISTS entry_state (
|
||||||
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||||
@@ -143,8 +139,8 @@ pub struct Managed {
|
|||||||
pub orphaned: bool,
|
pub orphaned: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Adds columns that later versions introduced. CREATE TABLE IF NOT EXISTS does nothing to
|
/// Adds the columns later versions introduced and drops the ones they retired. CREATE TABLE IF
|
||||||
/// a table that already exists, so an installed database needs them added explicitly.
|
/// NOT EXISTS does nothing to a table that already exists, so an installed database needs both.
|
||||||
fn migrate(conn: &Connection) -> Result<()> {
|
fn migrate(conn: &Connection) -> Result<()> {
|
||||||
let wanted: &[(&str, &str, &str)] = &[
|
let wanted: &[(&str, &str, &str)] = &[
|
||||||
("feeds", "image", "TEXT"),
|
("feeds", "image", "TEXT"),
|
||||||
@@ -155,18 +151,29 @@ fn migrate(conn: &Connection) -> Result<()> {
|
|||||||
("entries", "duration", "INTEGER"),
|
("entries", "duration", "INTEGER"),
|
||||||
("entries", "episode", "INTEGER"),
|
("entries", "episode", "INTEGER"),
|
||||||
("entries", "season", "INTEGER"),
|
("entries", "season", "INTEGER"),
|
||||||
("entries", "position", "INTEGER NOT NULL DEFAULT 0"),
|
|
||||||
];
|
];
|
||||||
for (table, column, ty) in wanted {
|
// Read state from before accounts, long since moved to entry_state. Two bugs came from
|
||||||
|
// queries still reading these after they stopped meaning anything, so they go.
|
||||||
|
let retired: &[(&str, &str)] = &[("entries", "read"), ("entries", "flagged"), ("entries", "position")];
|
||||||
|
let has = |table: &str, column: &str| -> Result<bool> {
|
||||||
let mut stmt = conn.prepare(&format!("PRAGMA table_info({table})"))?;
|
let mut stmt = conn.prepare(&format!("PRAGMA table_info({table})"))?;
|
||||||
let existing: Vec<String> = stmt
|
let names = stmt
|
||||||
.query_map([], |r| r.get::<_, String>(1))?
|
.query_map([], |r| r.get::<_, String>(1))?
|
||||||
.collect::<rusqlite::Result<Vec<_>>>()?;
|
.collect::<rusqlite::Result<Vec<_>>>()?;
|
||||||
if !existing.iter().any(|c| c == column) {
|
Ok(names.iter().any(|c| c == column))
|
||||||
|
};
|
||||||
|
for (table, column, ty) in wanted {
|
||||||
|
if !has(table, column)? {
|
||||||
tracing::info!(table, column, "adding column");
|
tracing::info!(table, column, "adding column");
|
||||||
conn.execute_batch(&format!("ALTER TABLE {table} ADD COLUMN {column} {ty}"))?;
|
conn.execute_batch(&format!("ALTER TABLE {table} ADD COLUMN {column} {ty}"))?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for (table, column) in retired {
|
||||||
|
if has(table, column)? {
|
||||||
|
tracing::info!(table, column, "dropping column");
|
||||||
|
conn.execute_batch(&format!("ALTER TABLE {table} DROP COLUMN {column}"))?;
|
||||||
|
}
|
||||||
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -345,22 +352,19 @@ impl Db {
|
|||||||
|
|
||||||
/// Returns true when this entry had not been seen before.
|
/// Returns true when this entry had not been seen before.
|
||||||
///
|
///
|
||||||
/// A changed description or title flips `read` back to 0, which is what the original's
|
|
||||||
/// textDiff dance was ultimately for -- minus the diff markup, which the UI can do.
|
|
||||||
pub fn record_entry(&self, feed_id: &str, e: &crate::feed::Entry) -> Result<bool> {
|
pub fn record_entry(&self, feed_id: &str, e: &crate::feed::Entry) -> Result<bool> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let inserted = conn.execute(
|
let inserted = conn.execute(
|
||||||
"INSERT OR IGNORE INTO entries
|
"INSERT OR IGNORE INTO entries
|
||||||
(feed_id, guid, title, link, published, description, first_seen, read, flagged,
|
(feed_id, guid, title, link, published, description, first_seen,
|
||||||
image, duration, episode, season)
|
image, duration, episode, season)
|
||||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0, 0, ?8, ?9, ?10, ?11)",
|
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)",
|
||||||
rusqlite::params![
|
rusqlite::params![
|
||||||
feed_id, e.guid, e.title, e.link, e.published, e.description, now(),
|
feed_id, e.guid, e.title, e.link, e.published, e.description, now(),
|
||||||
e.image, e.duration, e.episode, e.season
|
e.image, e.duration, e.episode, e.season
|
||||||
],
|
],
|
||||||
)?;
|
)?;
|
||||||
if inserted == 0 {
|
if inserted == 0 {
|
||||||
// The SET expressions see the pre-update row, so this compares old vs new.
|
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"UPDATE entries SET
|
"UPDATE entries SET
|
||||||
title = coalesce(?3, title),
|
title = coalesce(?3, title),
|
||||||
@@ -368,8 +372,7 @@ impl Db {
|
|||||||
image = coalesce(?5, image),
|
image = coalesce(?5, image),
|
||||||
duration = coalesce(?6, duration),
|
duration = coalesce(?6, duration),
|
||||||
episode = coalesce(?7, episode),
|
episode = coalesce(?7, episode),
|
||||||
season = coalesce(?8, season),
|
season = coalesce(?8, season)
|
||||||
read = CASE WHEN description IS NOT ?4 OR title IS NOT ?3 THEN 0 ELSE read END
|
|
||||||
WHERE feed_id = ?1 AND guid = ?2",
|
WHERE feed_id = ?1 AND guid = ?2",
|
||||||
rusqlite::params![
|
rusqlite::params![
|
||||||
feed_id, e.guid, e.title, e.description,
|
feed_id, e.guid, e.title, e.description,
|
||||||
@@ -687,22 +690,9 @@ pub struct EncRow {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl Db {
|
impl Db {
|
||||||
/// One page of a feed's entries, newest first, each with its enclosures attached.
|
/// One page of entries, each with its enclosures attached: one feed's, or every feed the
|
||||||
/// `search` matches title and description, case-insensitively.
|
/// person subscribes to when `feed_id` is None (All Subscriptions). `search` matches title
|
||||||
pub fn entries(
|
/// and description, case-insensitively.
|
||||||
&self,
|
|
||||||
user_id: i64,
|
|
||||||
feed_id: &str,
|
|
||||||
filter: Filter,
|
|
||||||
search: Option<&str>,
|
|
||||||
offset: i64,
|
|
||||||
limit: i64,
|
|
||||||
) -> Result<Vec<EntryRow>> {
|
|
||||||
self.entries_in(user_id, Some(feed_id), filter, search, offset, limit, &order_sql("published", "desc"))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `entries` for one feed, or across every feed the person subscribes to when `feed_id`
|
|
||||||
/// is None: the All Subscriptions view.
|
|
||||||
pub fn entries_in(
|
pub fn entries_in(
|
||||||
&self,
|
&self,
|
||||||
user_id: i64,
|
user_id: i64,
|
||||||
@@ -793,18 +783,7 @@ impl Db {
|
|||||||
Ok(rows)
|
Ok(rows)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// How many entries match, so the UI knows whether there is another page.
|
/// How many entries `entries_in` would page through, so the UI knows whether there is more.
|
||||||
pub fn count_entries(
|
|
||||||
&self,
|
|
||||||
user_id: i64,
|
|
||||||
feed_id: &str,
|
|
||||||
filter: Filter,
|
|
||||||
search: Option<&str>,
|
|
||||||
) -> Result<i64> {
|
|
||||||
self.count_in(user_id, Some(feed_id), filter, search)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `count_entries` for one feed, or across every feed the person subscribes to.
|
|
||||||
pub fn count_in(
|
pub fn count_in(
|
||||||
&self,
|
&self,
|
||||||
user_id: i64,
|
user_id: i64,
|
||||||
@@ -838,24 +817,16 @@ impl Db {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Marks every entry in a feed read, for the "mark all read" button.
|
/// The first admin starts subscribed to the whole catalogue: whoever wrote config.toml meant
|
||||||
/// Moves a single-user library onto an account: everything read, starred or part-played
|
/// to read those feeds, and without this a fresh install signs in to an empty sidebar. Runs
|
||||||
/// becomes that person's, and they subscribe to every feed already in the catalogue.
|
/// only while nobody subscribes to anything, so an unsubscribe is never undone.
|
||||||
/// Runs once -- the moment there is a first account and no subscriptions yet.
|
pub fn adopt_catalogue(&self, user_id: i64, catalogue: &[String]) -> Result<usize> {
|
||||||
pub fn adopt_existing_library(&self, user_id: i64, catalogue: &[String]) -> Result<usize> {
|
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let already: i64 =
|
let already: i64 =
|
||||||
conn.query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0))?;
|
conn.query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0))?;
|
||||||
if already > 0 {
|
if already > 0 {
|
||||||
return Ok(0);
|
return Ok(0);
|
||||||
}
|
}
|
||||||
let moved = conn.execute(
|
|
||||||
"INSERT INTO entry_state (user_id, feed_id, guid, read, flagged, position)
|
|
||||||
SELECT ?1, feed_id, guid, read, flagged, position FROM entries
|
|
||||||
WHERE read = 1 OR flagged = 1 OR position > 0
|
|
||||||
ON CONFLICT(user_id, feed_id, guid) DO NOTHING",
|
|
||||||
[user_id],
|
|
||||||
)?;
|
|
||||||
for id in catalogue {
|
for id in catalogue {
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created) VALUES (?1, ?2, ?3)",
|
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created) VALUES (?1, ?2, ?3)",
|
||||||
@@ -868,7 +839,7 @@ impl Db {
|
|||||||
SELECT ?1, id, ?2 FROM feeds",
|
SELECT ?1, id, ?2 FROM feeds",
|
||||||
params![user_id, now()],
|
params![user_id, now()],
|
||||||
)?;
|
)?;
|
||||||
Ok(moved)
|
Ok(catalogue.len())
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- subscriptions ----
|
// ---- subscriptions ----
|
||||||
@@ -920,14 +891,20 @@ impl Db {
|
|||||||
|
|
||||||
/// Everyone's settings for one feed. The scanner merges these into what it fetches
|
/// Everyone's settings for one feed. The scanner merges these into what it fetches
|
||||||
/// and downloads, since one file serves the lot.
|
/// and downloads, since one file serves the lot.
|
||||||
pub fn subscribers(&self, feed_id: &str) -> Result<Vec<Sub>> {
|
/// In a group, whatever someone has not set on the feed itself comes from their
|
||||||
|
/// subscription to the group, as the group's settings dialog has always said it does.
|
||||||
|
pub fn subscribers(&self, feed_id: &str, group: Option<&str>) -> Result<Vec<Sub>> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let mut stmt = conn.prepare(
|
let mut stmt = conn.prepare(
|
||||||
"SELECT keywords, auto_download, allow_explicit, max_new_per_check
|
"SELECT coalesce(c.keywords, p.keywords), coalesce(c.auto_download, p.auto_download),
|
||||||
FROM subscriptions WHERE feed_id = ?1",
|
coalesce(c.allow_explicit, p.allow_explicit),
|
||||||
|
coalesce(c.max_new_per_check, p.max_new_per_check)
|
||||||
|
FROM subscriptions c
|
||||||
|
LEFT JOIN subscriptions p ON p.user_id = c.user_id AND p.feed_id = ?2
|
||||||
|
WHERE c.feed_id = ?1",
|
||||||
)?;
|
)?;
|
||||||
let out = stmt
|
let out = stmt
|
||||||
.query_map([feed_id], |r| {
|
.query_map(params![feed_id, group], |r| {
|
||||||
Ok(Sub {
|
Ok(Sub {
|
||||||
feed_id: feed_id.to_string(),
|
feed_id: feed_id.to_string(),
|
||||||
keywords: r
|
keywords: r
|
||||||
@@ -1325,6 +1302,71 @@ impl Db {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Empties a feed of its items, leaving its files alone.
|
||||||
|
pub fn clear_entries(&self, feed_id: &str) -> Result<()> {
|
||||||
|
let conn = self.conn.lock().unwrap();
|
||||||
|
conn.execute("DELETE FROM entries WHERE feed_id = ?1", [feed_id])?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every feed the database holds rows for, as (id, url), removed ones included.
|
||||||
|
pub fn feed_urls(&self) -> Result<Vec<(String, String)>> {
|
||||||
|
let conn = self.conn.lock().unwrap();
|
||||||
|
let mut stmt = conn.prepare("SELECT id, coalesce(url, '') FROM feeds")?;
|
||||||
|
let out = stmt
|
||||||
|
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))?
|
||||||
|
.collect::<rusqlite::Result<_>>()?;
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hands a feed in a group the enclosures its parent holds, as (guid, url), with everyone's
|
||||||
|
/// read state for them. A Patreon creator read as one feed before it was split into shows
|
||||||
|
/// owns every show's files, and `enclosures.url` is unique, so without this each show would
|
||||||
|
/// list its items with nothing to play.
|
||||||
|
pub fn adopt(&self, parent: &str, child: &str, listed: &[(&str, &str)]) -> Result<()> {
|
||||||
|
let mut conn = self.conn.lock().unwrap();
|
||||||
|
let holds: bool = conn.query_row(
|
||||||
|
"SELECT EXISTS (SELECT 1 FROM enclosures WHERE feed_id = ?1)",
|
||||||
|
[parent],
|
||||||
|
|r| r.get(0),
|
||||||
|
)?;
|
||||||
|
if !holds {
|
||||||
|
return Ok(()); // An OPML, or a creator already shared out.
|
||||||
|
}
|
||||||
|
let tx = conn.transaction()?;
|
||||||
|
for &(guid, url) in listed {
|
||||||
|
let moved = tx.execute(
|
||||||
|
"UPDATE enclosures SET feed_id = ?3, guid = ?4 WHERE url = ?1 AND feed_id = ?2",
|
||||||
|
params![url, parent, child, guid],
|
||||||
|
)?;
|
||||||
|
if moved == 1 {
|
||||||
|
// Patreon gives a post the same guid in every feed it appears in.
|
||||||
|
tx.execute(
|
||||||
|
"UPDATE OR IGNORE entry_state SET feed_id = ?2 WHERE feed_id = ?1 AND guid = ?3",
|
||||||
|
params![parent, child, guid],
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tx.commit()?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A feed's enclosures skipped by one of its filters, by URL, with the reason: the verdicts a
|
||||||
|
/// change of settings can overturn. A torrent held back while torrents are off is not a
|
||||||
|
/// filter's call.
|
||||||
|
pub fn skipped_by_filter(&self, feed_id: &str) -> Result<std::collections::HashMap<String, String>> {
|
||||||
|
let conn = self.conn.lock().unwrap();
|
||||||
|
let mut stmt = conn.prepare(
|
||||||
|
"SELECT url, last_error FROM enclosures
|
||||||
|
WHERE feed_id = ?1 AND state = 'skipped' AND last_error IS NOT NULL
|
||||||
|
AND last_error != 'torrents disabled'",
|
||||||
|
)?;
|
||||||
|
let out = stmt
|
||||||
|
.query_map([feed_id], |r| Ok((r.get(0)?, r.get(1)?)))?
|
||||||
|
.collect::<rusqlite::Result<_>>()?;
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn set_orphaned(&self, feed_id: &str, on: bool) -> Result<()> {
|
pub fn set_orphaned(&self, feed_id: &str, on: bool) -> Result<()> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
conn.execute(
|
conn.execute(
|
||||||
@@ -1457,8 +1499,9 @@ mod tests {
|
|||||||
// Starring and position are just as private.
|
// Starring and position are just as private.
|
||||||
db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).unwrap();
|
db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).unwrap();
|
||||||
db.set_position(2, "f", "b", 42).unwrap();
|
db.set_position(2, "f", "b", 42).unwrap();
|
||||||
let ray = db.entries(1, "f", Filter::All, None, 0, 50).unwrap();
|
let order = order_sql("published", "desc");
|
||||||
let sam = db.entries(2, "f", Filter::All, None, 0, 50).unwrap();
|
let page = |user| db.entries_in(user, Some("f"), Filter::All, None, 0, 50, &order).unwrap();
|
||||||
|
let (ray, sam) = (page(1), page(2));
|
||||||
let ray_b = ray.iter().find(|e| e.guid == "b").unwrap();
|
let ray_b = ray.iter().find(|e| e.guid == "b").unwrap();
|
||||||
let sam_b = sam.iter().find(|e| e.guid == "b").unwrap();
|
let sam_b = sam.iter().find(|e| e.guid == "b").unwrap();
|
||||||
assert!(ray_b.flagged && ray_b.position == 0);
|
assert!(ray_b.flagged && ray_b.position == 0);
|
||||||
@@ -1482,6 +1525,49 @@ mod tests {
|
|||||||
assert_eq!(sum.downloaded, 0);
|
assert_eq!(sum.downloaded, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_old_database_loses_the_retired_read_columns() {
|
||||||
|
let conn = Connection::open_in_memory().unwrap();
|
||||||
|
conn.execute_batch(
|
||||||
|
"CREATE TABLE entries (feed_id TEXT NOT NULL, guid TEXT NOT NULL,
|
||||||
|
first_seen INTEGER NOT NULL, read INTEGER NOT NULL DEFAULT 0,
|
||||||
|
flagged INTEGER NOT NULL DEFAULT 0, position INTEGER NOT NULL DEFAULT 0,
|
||||||
|
PRIMARY KEY (feed_id, guid));",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
// The same order as open(): the schema leaves the old table alone, migrate() fixes it.
|
||||||
|
conn.execute_batch(SCHEMA).unwrap();
|
||||||
|
migrate(&conn).unwrap();
|
||||||
|
let cols: Vec<String> = conn
|
||||||
|
.prepare("PRAGMA table_info(entries)")
|
||||||
|
.unwrap()
|
||||||
|
.query_map([], |r| r.get(1))
|
||||||
|
.unwrap()
|
||||||
|
.collect::<rusqlite::Result<_>>()
|
||||||
|
.unwrap();
|
||||||
|
assert!(!cols.iter().any(|c| ["read", "flagged", "position"].contains(&c.as_str())), "{cols:?}");
|
||||||
|
assert!(cols.iter().any(|c| c == "image"), "and it still gains the newer ones");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_first_admin_starts_with_the_catalogue_and_only_once() {
|
||||||
|
// Cutting this along with the dead read columns left the browser suite's admin with an
|
||||||
|
// empty sidebar: it is how a fresh install's first account gets config.toml's feeds.
|
||||||
|
let db = Db::memory().unwrap();
|
||||||
|
db.exec_for_test("INSERT INTO users (id, name, is_admin, created) VALUES (1,'admin',1,0);")
|
||||||
|
.unwrap();
|
||||||
|
let subs = || -> i64 {
|
||||||
|
db.conn.lock().unwrap().query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0)).unwrap()
|
||||||
|
};
|
||||||
|
let catalogue = ["a".to_string(), "b".to_string()];
|
||||||
|
assert_eq!(db.adopt_catalogue(1, &catalogue).unwrap(), 2);
|
||||||
|
assert_eq!(subs(), 2);
|
||||||
|
// Once anyone subscribes to anything it never runs again, so an unsubscribe sticks.
|
||||||
|
db.unsubscribe(1, "a").unwrap();
|
||||||
|
assert_eq!(db.adopt_catalogue(1, &catalogue).unwrap(), 0);
|
||||||
|
assert_eq!(subs(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn every_filter_works_with_and_without_a_search_term() {
|
fn every_filter_works_with_and_without_a_search_term() {
|
||||||
// Regression: the search clause used to be omitted when no term was given, while
|
// Regression: the search clause used to be omitted when no term was given, while
|
||||||
@@ -1505,21 +1591,22 @@ mod tests {
|
|||||||
|
|
||||||
for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged] {
|
for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged] {
|
||||||
// Both paths must run without erroring, and agree with each other.
|
// Both paths must run without erroring, and agree with each other.
|
||||||
let rows = db.entries(7, "f", f, None, 0, 50).unwrap();
|
let order = order_sql("published", "desc");
|
||||||
let n = db.count_entries(7, "f", f, None).unwrap();
|
let rows = db.entries_in(7, Some("f"), f, None, 0, 50, &order).unwrap();
|
||||||
|
let n = db.count_in(7, Some("f"), f, None).unwrap();
|
||||||
assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page");
|
assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page");
|
||||||
|
|
||||||
let rows = db.entries(7, "f", f, Some("dive"), 0, 50).unwrap();
|
let rows = db.entries_in(7, Some("f"), f, Some("dive"), 0, 50, &order).unwrap();
|
||||||
let n = db.count_entries(7, "f", f, Some("dive")).unwrap();
|
let n = db.count_in(7, Some("f"), f, Some("dive")).unwrap();
|
||||||
assert_eq!(rows.len() as i64, n, "{f:?} with search disagrees");
|
assert_eq!(rows.len() as i64, n, "{f:?} with search disagrees");
|
||||||
}
|
}
|
||||||
|
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::All, None).unwrap(), 3);
|
assert_eq!(db.count_in(7, Some("f"), Filter::All, None).unwrap(), 3);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::Unread, None).unwrap(), 1);
|
assert_eq!(db.count_in(7, Some("f"), Filter::Unread, None).unwrap(), 1);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::Downloaded, None).unwrap(), 1);
|
assert_eq!(db.count_in(7, Some("f"), Filter::Downloaded, None).unwrap(), 1);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::Flagged, None).unwrap(), 1);
|
assert_eq!(db.count_in(7, Some("f"), Filter::Flagged, None).unwrap(), 1);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::All, Some("dive")).unwrap(), 2);
|
assert_eq!(db.count_in(7, Some("f"), Filter::All, Some("dive")).unwrap(), 2);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::All, Some("NOTES two")).unwrap(), 1,
|
assert_eq!(db.count_in(7, Some("f"), Filter::All, Some("NOTES two")).unwrap(), 1,
|
||||||
"search is case-insensitive and covers the description");
|
"search is case-insensitive and covers the description");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1561,6 +1648,61 @@ mod tests {
|
|||||||
assert_eq!(state(4), "done");
|
assert_eq!(state(4), "done");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_show_takes_over_what_its_creator_held() {
|
||||||
|
let db = Db::memory().unwrap();
|
||||||
|
db.exec_for_test(
|
||||||
|
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0);
|
||||||
|
INSERT INTO enclosures (id, feed_id, guid, url, state, path, last_error) VALUES
|
||||||
|
(1,'creator','a','u1','done','/x/a.mp3',NULL),
|
||||||
|
(2,'creator','b','u2','skipped',NULL,'explicit'),
|
||||||
|
(3,'creator','c','u3','skipped',NULL,'explicit'),
|
||||||
|
(4,'other','d','u4','skipped',NULL,'torrents disabled');
|
||||||
|
INSERT INTO entry_state (user_id, feed_id, guid, read) VALUES (1,'creator','a',1);",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
db.adopt("creator", "show", &[("a", "u1"), ("b", "u2"), ("d", "u4")]).unwrap();
|
||||||
|
{
|
||||||
|
let conn = db.conn.lock().unwrap();
|
||||||
|
let owner = |id: i64| -> String {
|
||||||
|
conn.query_row("SELECT feed_id FROM enclosures WHERE id = ?1", [id], |r| r.get(0)).unwrap()
|
||||||
|
};
|
||||||
|
assert_eq!(owner(1), "show", "a downloaded file moves with its item");
|
||||||
|
assert_eq!(owner(2), "show");
|
||||||
|
assert_eq!(owner(3), "creator", "this show does not list it");
|
||||||
|
assert_eq!(owner(4), "other", "only the parent's are taken");
|
||||||
|
let read: String = conn
|
||||||
|
.query_row("SELECT feed_id FROM entry_state WHERE user_id = 1 AND guid = 'a'", [], |r| r.get(0))
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(read, "show", "what you had read stays read");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only a filter's verdict can be overturned by a change of settings.
|
||||||
|
let skipped = db.skipped_by_filter("show").unwrap();
|
||||||
|
assert_eq!(skipped.get("u2").map(String::as_str), Some("explicit"));
|
||||||
|
assert_eq!(skipped.len(), 1);
|
||||||
|
assert!(db.skipped_by_filter("other").unwrap().is_empty(), "torrents disabled is not a filter");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_feed_in_a_group_follows_your_settings_on_the_group() {
|
||||||
|
let db = Db::memory().unwrap();
|
||||||
|
db.exec_for_test(
|
||||||
|
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0);
|
||||||
|
INSERT INTO subscriptions (user_id, feed_id, allow_explicit, created) VALUES
|
||||||
|
(1,'group',1,0),(1,'show',NULL,0),(2,'group',1,0),(2,'show',0,0);",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let explicit = |group| -> Vec<Option<bool>> {
|
||||||
|
let mut v: Vec<_> =
|
||||||
|
db.subscribers("show", group).unwrap().into_iter().map(|s| s.allow_explicit).collect();
|
||||||
|
v.sort();
|
||||||
|
v
|
||||||
|
};
|
||||||
|
assert_eq!(explicit(Some("group")), [Some(false), Some(true)], "ray inherits; sam's own choice on the show wins");
|
||||||
|
assert_eq!(explicit(None), [None, Some(false)], "outside a group nothing is inherited");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn enclosure_url_is_the_dedupe_key() {
|
fn enclosure_url_is_the_dedupe_key() {
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().unwrap();
|
||||||
|
|||||||
@@ -223,7 +223,7 @@ enum Sniffed {
|
|||||||
/// 2008 and so always answered 'data'.
|
/// 2008 and so always answered 'data'.
|
||||||
async fn sniff(path: &Path) -> Result<Sniffed> {
|
async fn sniff(path: &Path) -> Result<Sniffed> {
|
||||||
let head = read_head(path, 512).await?;
|
let head = read_head(path, 512).await?;
|
||||||
if infer::is(&head, "torrent") || head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
|
if head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
|
||||||
return Ok(Sniffed::Torrent);
|
return Ok(Sniffed::Torrent);
|
||||||
}
|
}
|
||||||
let text = String::from_utf8_lossy(&head);
|
let text = String::from_utf8_lossy(&head);
|
||||||
|
|||||||
136
src/feed.rs
136
src/feed.rs
@@ -117,6 +117,103 @@ pub fn opml_title(bytes: &[u8]) -> Option<String> {
|
|||||||
.filter(|t| !t.is_empty())
|
.filter(|t| !t.is_empty())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The token and show of a Patreon feed link, or None for any other URL.
|
||||||
|
///
|
||||||
|
/// Patreon gives each patron one token per creator. With no show it stands for the creator,
|
||||||
|
/// whose feed carries every show at once.
|
||||||
|
fn patreon_parts(url: &str) -> Option<(String, Option<String>)> {
|
||||||
|
let u = url::Url::parse(url).ok()?;
|
||||||
|
if !matches!(u.host_str()?, "patreon.com" | "www.patreon.com") || !u.path().starts_with("/rss") {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let param = |name: &str| u.query_pairs().find(|(k, _)| k == name).map(|(_, v)| v.into_owned());
|
||||||
|
Some((param("auth")?, param("show")))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A Patreon link naming a creator but no show.
|
||||||
|
pub fn is_patreon_creator(url: &str) -> bool {
|
||||||
|
matches!(patreon_parts(url), Some((_, None)))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What was typed into Add feed, as a URL. A bare Patreon token is taken as its creator's
|
||||||
|
/// feed, since the token alone says whose it is.
|
||||||
|
pub fn expand_input(input: &str) -> String {
|
||||||
|
let s = input.trim();
|
||||||
|
let token = s.len() >= 20 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_');
|
||||||
|
if token { format!("https://www.patreon.com/rss?auth={s}") } else { s.to_owned() }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether two URLs are the same feed. One Patreon show has several spellings -- by the
|
||||||
|
/// creator's name, by number, or with no creator at all -- and the token and show are what
|
||||||
|
/// identify it.
|
||||||
|
pub fn same_feed(a: &str, b: &str) -> bool {
|
||||||
|
a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A Patreon creator's name and shows, each show as (title, feed URL).
|
||||||
|
///
|
||||||
|
/// ponytail: Patreon's own web API, undocumented, asked without signing in. If it changes,
|
||||||
|
/// finding shows stops and the show feeds already found keep working. The documented API
|
||||||
|
/// needs an OAuth client per install and does not list shows.
|
||||||
|
pub async fn patreon_shows(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
url: &str,
|
||||||
|
) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||||
|
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
|
||||||
|
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
|
||||||
|
let mut resp = client.get(url).send().await.context("connecting")?;
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
|
||||||
|
}
|
||||||
|
let mut head = Vec::new();
|
||||||
|
while patreon_campaign(&head).is_none() && head.len() < 64 * 1024 {
|
||||||
|
let Some(chunk) = resp.chunk().await.context("reading the feed")? else { break };
|
||||||
|
head.extend_from_slice(&chunk);
|
||||||
|
}
|
||||||
|
let campaign = patreon_campaign(&head)
|
||||||
|
.ok_or_else(|| anyhow!("the Patreon feed does not say whose it is"))?;
|
||||||
|
|
||||||
|
let api = format!(
|
||||||
|
"https://www.patreon.com/api/campaigns/{campaign}\
|
||||||
|
?include=shows&fields%5Bcampaign%5D=name&fields%5Bcollection%5D=title"
|
||||||
|
);
|
||||||
|
let resp = client.get(api).send().await.context("asking Patreon for the shows")?;
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
return Err(anyhow!("Patreon would not list the shows: HTTP {}", resp.status()));
|
||||||
|
}
|
||||||
|
let (name, shows) = parse_patreon_shows(&resp.bytes().await.context("reading the shows")?)?;
|
||||||
|
Ok((name, shows.into_iter().map(|(id, title)| (title, format!("{url}&show={id}"))).collect()))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The campaign number in the start of a Patreon feed.
|
||||||
|
fn patreon_campaign(head: &[u8]) -> Option<String> {
|
||||||
|
let text = String::from_utf8_lossy(head);
|
||||||
|
text.match_indices("patreon.com/rss/").find_map(|(i, m)| {
|
||||||
|
let id: String = text[i + m.len()..].chars().take_while(char::is_ascii_digit).collect();
|
||||||
|
(!id.is_empty()).then_some(id)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A campaign's name and its shows as (id, title), from Patreon's JSON:API answer.
|
||||||
|
fn parse_patreon_shows(json: &[u8]) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||||
|
let v: serde_json::Value = serde_json::from_slice(json).context("Patreon's answer is not JSON")?;
|
||||||
|
// Missing is not the same as none. Read as no shows, the creator feed would be scanned as
|
||||||
|
// a plain feed, claim every show's files, and leave the shows empty once the list returned.
|
||||||
|
let ids = v["data"]["relationships"]["shows"]["data"]
|
||||||
|
.as_array()
|
||||||
|
.ok_or_else(|| anyhow!("Patreon's answer does not list the shows"))?;
|
||||||
|
let title = |id: &str| -> Option<String> {
|
||||||
|
let show = v["included"].as_array()?.iter().find(|x| x["type"] == "collection" && x["id"] == id)?;
|
||||||
|
show["attributes"]["title"].as_str().map(|t| t.trim().to_owned())
|
||||||
|
};
|
||||||
|
let shows = ids
|
||||||
|
.iter()
|
||||||
|
.filter_map(|s| s["id"].as_str())
|
||||||
|
.map(|id| (id.to_owned(), title(id).unwrap_or_else(|| format!("Show {id}"))))
|
||||||
|
.collect();
|
||||||
|
Ok((v["data"]["attributes"]["name"].as_str().map(str::to_owned), shows))
|
||||||
|
}
|
||||||
|
|
||||||
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
|
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
|
||||||
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
|
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
|
||||||
match rss::Channel::read_from(bytes) {
|
match rss::Channel::read_from(bytes) {
|
||||||
@@ -550,6 +647,45 @@ mod tests {
|
|||||||
assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
|
assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_patreon_creator_is_a_list_of_its_shows() {
|
||||||
|
let tok = "AbCdEfGhIjKlMnOpQrStUvWxYz012_-9";
|
||||||
|
assert_eq!(expand_input(&format!(" {tok} ")), format!("https://www.patreon.com/rss?auth={tok}"));
|
||||||
|
assert_eq!(expand_input("https://example.com/rss"), "https://example.com/rss");
|
||||||
|
|
||||||
|
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss/glasscannon?auth={tok}")));
|
||||||
|
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss?auth={tok}")));
|
||||||
|
assert!(!is_patreon_creator(&format!("https://www.patreon.com/rss/x?auth={tok}&show=1")), "one show is a feed");
|
||||||
|
assert!(!is_patreon_creator(&format!("https://example.com/rss?auth={tok}")));
|
||||||
|
|
||||||
|
// The show you already have by name is the one a bare token would add by number.
|
||||||
|
assert!(same_feed(
|
||||||
|
&format!("https://www.patreon.com/rss/glasscannon?auth={tok}&show=2073588"),
|
||||||
|
&format!("https://www.patreon.com/rss?auth={tok}&show=2073588"),
|
||||||
|
));
|
||||||
|
assert!(!same_feed(
|
||||||
|
&format!("https://www.patreon.com/rss?auth={tok}&show=1"),
|
||||||
|
&format!("https://www.patreon.com/rss?auth={tok}&show=2"),
|
||||||
|
));
|
||||||
|
|
||||||
|
// The self link carries the campaign by number, whichever spelling was asked for.
|
||||||
|
let head = br#"<rss><channel><link>https://www.patreon.com/glasscannon</link>
|
||||||
|
<atom:link href="https://www.patreon.com/rss/369921?auth=t" rel="self"/>"#;
|
||||||
|
assert_eq!(patreon_campaign(head).as_deref(), Some("369921"));
|
||||||
|
assert_eq!(patreon_campaign(b"<rss><channel><title>T"), None);
|
||||||
|
|
||||||
|
let json = br#"{"data":{"id":"369921","type":"campaign","attributes":{"name":"The Glass Cannon Network"},
|
||||||
|
"relationships":{"shows":{"data":[{"id":"2073588","type":"collection"},{"id":"2073636","type":"collection"}]}}},
|
||||||
|
"included":[{"id":"2073588","type":"collection","attributes":{"title":"Get in the Trunk "}},
|
||||||
|
{"id":"2073636","type":"collection","attributes":{"title":"Shadowdark"}}]}"#;
|
||||||
|
let (name, shows) = parse_patreon_shows(json).unwrap();
|
||||||
|
assert_eq!(name.as_deref(), Some("The Glass Cannon Network"));
|
||||||
|
assert_eq!(shows, [("2073588".into(), "Get in the Trunk".into()), ("2073636".into(), "Shadowdark".into())]);
|
||||||
|
|
||||||
|
// An answer that stops naming the shows is an error, never "this creator has none".
|
||||||
|
assert!(parse_patreon_shows(br#"{"data":{"attributes":{"name":"X"}}}"#).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn an_item_may_carry_several_enclosures() {
|
fn an_item_may_carry_several_enclosures() {
|
||||||
// The rss crate keeps only one per item -- the last -- so these come from the XML.
|
// The rss crate keeps only one per item -- the last -- so these come from the XML.
|
||||||
|
|||||||
@@ -111,18 +111,11 @@ impl Visit for Collect {
|
|||||||
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
||||||
self.add(field, format!("{value:?}"));
|
self.add(field, format!("{value:?}"));
|
||||||
}
|
}
|
||||||
|
// Numbers and bools reach record_debug through the trait's defaults, which prints them the
|
||||||
|
// same way. A string would print quoted there, hence its own method.
|
||||||
fn record_str(&mut self, field: &Field, value: &str) {
|
fn record_str(&mut self, field: &Field, value: &str) {
|
||||||
self.add(field, value.to_owned());
|
self.add(field, value.to_owned());
|
||||||
}
|
}
|
||||||
fn record_i64(&mut self, field: &Field, value: i64) {
|
|
||||||
self.add(field, value.to_string());
|
|
||||||
}
|
|
||||||
fn record_u64(&mut self, field: &Field, value: u64) {
|
|
||||||
self.add(field, value.to_string());
|
|
||||||
}
|
|
||||||
fn record_bool(&mut self, field: &Field, value: bool) {
|
|
||||||
self.add(field, value.to_string());
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
|
|||||||
153
src/main.rs
153
src/main.rs
@@ -334,8 +334,7 @@ async fn daemon(
|
|||||||
anyhow::bail!("a daemon is already listening on {}", socket.display());
|
anyhow::bail!("a daemon is already listening on {}", socket.display());
|
||||||
}
|
}
|
||||||
|
|
||||||
// A database with nobody in it cannot be signed into, and an install that predates
|
// A database with nobody in it cannot be signed into.
|
||||||
// accounts still has to serve its owner. Both get the same starting point.
|
|
||||||
if ctx.db.users()?.is_empty() {
|
if ctx.db.users()?.is_empty() {
|
||||||
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
|
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
|
||||||
tracing::warn!(
|
tracing::warn!(
|
||||||
@@ -344,22 +343,15 @@ async fn daemon(
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// A library that predates accounts belongs to whoever was using it: the admin.
|
|
||||||
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
|
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
|
||||||
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
|
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
|
||||||
match ctx.db.adopt_existing_library(admin.id, &catalogue) {
|
match ctx.db.adopt_catalogue(admin.id, &catalogue) {
|
||||||
Ok(0) => {}
|
Ok(0) => {}
|
||||||
Ok(n) => tracing::info!(user = %admin.name, entries = n, "adopted the existing library"),
|
Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
|
||||||
Err(e) => tracing::error!(error = %e, "could not adopt the existing library"),
|
Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
match migrate_opml_children(&ctx) {
|
|
||||||
Ok(n) if n > 0 => tracing::info!(count = n, "moved OPML feeds out of config.toml into the database"),
|
|
||||||
Ok(_) => {}
|
|
||||||
Err(e) => tracing::warn!(error = ?e, "could not tidy OPML feeds out of the config"),
|
|
||||||
}
|
|
||||||
|
|
||||||
match ctx.db.requeue_interrupted() {
|
match ctx.db.requeue_interrupted() {
|
||||||
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
|
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
|
||||||
Ok(_) => {}
|
Ok(_) => {}
|
||||||
@@ -468,7 +460,7 @@ async fn start_web(
|
|||||||
let mut fresh = (*cfg).clone();
|
let mut fresh = (*cfg).clone();
|
||||||
fresh.web.enabled = true;
|
fresh.web.enabled = true;
|
||||||
fresh.web.bind = bind.clone();
|
fresh.web.bind = bind.clone();
|
||||||
fresh.web.token = web::generate_token();
|
fresh.web.token = crate::auth::new_session_token();
|
||||||
fresh.save(config_path)?;
|
fresh.save(config_path)?;
|
||||||
ctx.reload_cfg(config_path)?;
|
ctx.reload_cfg(config_path)?;
|
||||||
println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token);
|
println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token);
|
||||||
@@ -517,8 +509,9 @@ async fn add(
|
|||||||
keywords: Vec<String>,
|
keywords: Vec<String>,
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
let mut cfg = (*ctx.cfg()).clone();
|
let mut cfg = (*ctx.cfg()).clone();
|
||||||
|
let url = &feed::expand_input(url);
|
||||||
// Includes feeds derived from an OPML, or the same show could be added twice.
|
// Includes feeds derived from an OPML, or the same show could be added twice.
|
||||||
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| s.cfg.url == url) {
|
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
|
||||||
anyhow::bail!("already subscribed as {:?}", existing.id);
|
anyhow::bail!("already subscribed as {:?}", existing.id);
|
||||||
}
|
}
|
||||||
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
|
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
|
||||||
@@ -569,10 +562,17 @@ pub async fn add_one(
|
|||||||
|
|
||||||
// Slugs must be unique across derived feeds too, or a new feed can collide with one
|
// Slugs must be unique across derived feeds too, or a new feed can collide with one
|
||||||
// an OPML already introduced.
|
// an OPML already introduced.
|
||||||
let taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
|
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.map(|s| (s.id, s.cfg))
|
.map(|s| (s.id, s.cfg))
|
||||||
.collect();
|
.collect();
|
||||||
|
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
|
||||||
|
// it gets its history back, and a different feed does not inherit someone else's.
|
||||||
|
for (id, other) in ctx.db.feed_urls()? {
|
||||||
|
if !feed::same_feed(&other, url) {
|
||||||
|
taken.entry(id).or_insert_with(|| probe.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
let id = config::unique_slug(&title, &taken);
|
let id = config::unique_slug(&title, &taken);
|
||||||
cfg.feeds.insert(id.clone(), probe);
|
cfg.feeds.insert(id.clone(), probe);
|
||||||
Ok(id)
|
Ok(id)
|
||||||
@@ -807,7 +807,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
|
|||||||
ctx.out.emit(Event::FeedSkip {
|
ctx.out.emit(Event::FeedSkip {
|
||||||
feed: id.clone(),
|
feed: id.clone(),
|
||||||
reason: format!(
|
reason: format!(
|
||||||
"OPML: {total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
|
"{total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
|
||||||
added.len()
|
added.len()
|
||||||
),
|
),
|
||||||
});
|
});
|
||||||
@@ -908,36 +908,6 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
|
|||||||
Ok(out)
|
Ok(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Moves OPML children that older versions wrote into config.toml over to the database.
|
|
||||||
/// They were never yours to edit, and 80-odd of them made the file unreadable.
|
|
||||||
fn migrate_opml_children(ctx: &Ctx) -> Result<usize> {
|
|
||||||
let cfg = (*ctx.cfg()).clone();
|
|
||||||
let children: Vec<(String, config::Feed)> = cfg
|
|
||||||
.feeds
|
|
||||||
.iter()
|
|
||||||
.filter(|(_, f)| f.group.is_some())
|
|
||||||
.map(|(id, f)| (id.clone(), f.clone()))
|
|
||||||
.collect();
|
|
||||||
if children.is_empty() {
|
|
||||||
return Ok(0);
|
|
||||||
}
|
|
||||||
let mut fresh = cfg.clone();
|
|
||||||
for (id, f) in &children {
|
|
||||||
let group = f.group.clone().unwrap_or_default();
|
|
||||||
let title = ctx
|
|
||||||
.db
|
|
||||||
.feed_summary(id)
|
|
||||||
.ok()
|
|
||||||
.and_then(|s| s.title)
|
|
||||||
.unwrap_or_else(|| id.clone());
|
|
||||||
ctx.db.upsert_managed(id, &f.url, &title, &group)?;
|
|
||||||
fresh.feeds.remove(id);
|
|
||||||
}
|
|
||||||
fresh.save(&ctx.config_path)?;
|
|
||||||
ctx.reload_cfg(&ctx.config_path)?;
|
|
||||||
Ok(children.len())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Seconds to wait before re-checking a feed.
|
/// Seconds to wait before re-checking a feed.
|
||||||
///
|
///
|
||||||
/// A per-feed schedule is an explicit instruction and wins outright. Without one, the
|
/// A per-feed schedule is an explicit instruction and wins outright. Without one, the
|
||||||
@@ -963,7 +933,7 @@ struct Scan {
|
|||||||
enum Outcome {
|
enum Outcome {
|
||||||
NotModified,
|
NotModified,
|
||||||
Feed(Scan),
|
Feed(Scan),
|
||||||
/// The URL served an OPML document, so it is a subscription list rather than a feed.
|
/// The URL is a list of feeds rather than a feed: an OPML, or a Patreon creator's shows.
|
||||||
Opml { added: Vec<String>, removed: usize, kept: usize, total: usize },
|
Opml { added: Vec<String>, removed: usize, kept: usize, total: usize },
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -973,6 +943,31 @@ async fn scan_one(
|
|||||||
feed_cfg: &config::Feed,
|
feed_cfg: &config::Feed,
|
||||||
state: &db::HttpState,
|
state: &db::HttpState,
|
||||||
) -> Result<Outcome> {
|
) -> Result<Outcome> {
|
||||||
|
// A Patreon creator with more than one show is a list of feeds, like an OPML.
|
||||||
|
if feed::is_patreon_creator(&feed_cfg.url) {
|
||||||
|
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
|
||||||
|
Ok((name, shows)) if shows.len() > 1 => {
|
||||||
|
ctx.db.touch_feed(id, &feed_cfg.url)?;
|
||||||
|
if let Some(name) = name {
|
||||||
|
ctx.db.set_title(id, &name)?;
|
||||||
|
}
|
||||||
|
// Read as one feed before it was split, it listed every show's items in one
|
||||||
|
// heap. The items go; its files and read state move to each show as the show
|
||||||
|
// lists them (`Db::adopt`), so no show comes up empty for want of a URL.
|
||||||
|
ctx.db.clear_entries(id)?;
|
||||||
|
return sync_group(ctx, id, feed_cfg, &shows).await;
|
||||||
|
}
|
||||||
|
Ok(_) => {} // One show: the creator's feed is that show.
|
||||||
|
// Already split: keep the shows it has rather than read the creator as one heap.
|
||||||
|
Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e),
|
||||||
|
Err(e) => tracing::warn!(
|
||||||
|
feed = id,
|
||||||
|
error = %format!("{e:#}"),
|
||||||
|
"could not list the Patreon shows; reading it as one feed"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
let mut fetched = feed::fetch(
|
let mut fetched = feed::fetch(
|
||||||
&ctx.client,
|
&ctx.client,
|
||||||
feed_cfg,
|
feed_cfg,
|
||||||
@@ -1018,19 +1013,38 @@ async fn scan_one(
|
|||||||
)?;
|
)?;
|
||||||
|
|
||||||
let policy = policy_for(ctx, id, feed_cfg)?;
|
let policy = policy_for(ctx, id, feed_cfg)?;
|
||||||
|
if let Some(parent) = &feed_cfg.group {
|
||||||
|
let listed: Vec<(&str, &str)> = parsed
|
||||||
|
.entries
|
||||||
|
.iter()
|
||||||
|
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
|
||||||
|
.collect();
|
||||||
|
ctx.db.adopt(parent, id, &listed)?;
|
||||||
|
}
|
||||||
|
// Verdicts are recorded in `state`, so the download queue below is just "everything still
|
||||||
|
// pending". A filter's verdict is looked at again on every scan, though: made once, at
|
||||||
|
// discovery, it outlived the setting behind it, and allowing explicit items afterwards
|
||||||
|
// changed nothing however often the feed was scanned.
|
||||||
|
let skipped = ctx.db.skipped_by_filter(id)?;
|
||||||
let mut scan = Scan::default();
|
let mut scan = Scan::default();
|
||||||
for entry in &parsed.entries {
|
for entry in &parsed.entries {
|
||||||
if ctx.db.record_entry(id, entry)? {
|
if ctx.db.record_entry(id, entry)? {
|
||||||
scan.new_entries += 1;
|
scan.new_entries += 1;
|
||||||
}
|
}
|
||||||
for enc in &entry.enclosures {
|
for enc in &entry.enclosures {
|
||||||
if !ctx.db.record_enclosure(id, &entry.guid, enc)? {
|
let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? {
|
||||||
continue; // Seen before: downloaded, skipped or deliberately reaped.
|
None
|
||||||
}
|
} else if let Some(reason) = skipped.get(&enc.url) {
|
||||||
// Filters run once, at discovery, and are recorded in `state`. The download
|
Some(reason.as_str())
|
||||||
// queue below is then just "everything still pending".
|
} else {
|
||||||
if let Some(reason) = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc) {
|
continue; // Settled: queued, downloaded, reaped, or another feed's file.
|
||||||
ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?;
|
};
|
||||||
|
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
|
||||||
|
if now != was {
|
||||||
|
match now {
|
||||||
|
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?,
|
||||||
|
None => ctx.db.mark_enclosure(&enc.url, "pending", None)?,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1114,12 +1128,6 @@ async fn scan_one(
|
|||||||
Ok(Outcome::Feed(scan))
|
Ok(Outcome::Feed(scan))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Brings the feed list in step with a subscribed OPML.
|
|
||||||
///
|
|
||||||
/// New entries are added under the OPML's group and folder. An entry that has gone from
|
|
||||||
/// the OPML is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
|
|
||||||
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
|
|
||||||
/// the UI to explain them.
|
|
||||||
async fn sync_opml(
|
async fn sync_opml(
|
||||||
ctx: &Arc<Ctx>,
|
ctx: &Arc<Ctx>,
|
||||||
parent_id: &str,
|
parent_id: &str,
|
||||||
@@ -1130,25 +1138,44 @@ async fn sync_opml(
|
|||||||
if let Some(title) = feed::opml_title(bytes) {
|
if let Some(title) = feed::opml_title(bytes) {
|
||||||
ctx.db.set_title(parent_id, &title)?;
|
ctx.db.set_title(parent_id, &title)?;
|
||||||
}
|
}
|
||||||
|
sync_group(ctx, parent_id, parent, &listed).await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Brings the feed list in step with a list of feeds: a subscribed OPML, or a Patreon
|
||||||
|
/// creator's shows.
|
||||||
|
///
|
||||||
|
/// New entries are added under the list's group and folder. An entry that has gone from
|
||||||
|
/// the list is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
|
||||||
|
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
|
||||||
|
/// the UI to explain them.
|
||||||
|
async fn sync_group(
|
||||||
|
ctx: &Arc<Ctx>,
|
||||||
|
parent_id: &str,
|
||||||
|
parent: &config::Feed,
|
||||||
|
listed: &[(String, String)],
|
||||||
|
) -> Result<Outcome> {
|
||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
let existing = ctx.db.managed_feeds()?;
|
let existing = ctx.db.managed_feeds()?;
|
||||||
let mut added = vec![];
|
let mut added = vec![];
|
||||||
|
|
||||||
for (title, url) in &listed {
|
for (title, url) in listed {
|
||||||
// Already known, whether derived or promoted into the config.
|
// Already known, whether derived or promoted into the config.
|
||||||
if let Some(m) = existing.iter().find(|m| &m.url == url) {
|
if let Some(m) = existing.iter().find(|m| &m.url == url) {
|
||||||
ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
|
ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if cfg.feeds.values().any(|f| &f.url == url) {
|
// A Patreon show you added by hand may be spelled differently from the one listed.
|
||||||
|
if cfg.feeds.values().any(|f| feed::same_feed(&f.url, url)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
// A removed feed keeps its rows, so its id is only free again for the same feed.
|
||||||
|
let known = ctx.db.feed_urls()?;
|
||||||
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
|
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
|
||||||
.feeds
|
.feeds
|
||||||
.keys()
|
.keys()
|
||||||
.chain(existing.iter().map(|m| &m.id))
|
.chain(existing.iter().map(|m| &m.id))
|
||||||
.chain(added.iter())
|
.chain(added.iter())
|
||||||
|
.chain(known.iter().filter(|(_, u)| !feed::same_feed(u, url)).map(|(id, _)| id))
|
||||||
.map(|id| (id.clone(), parent.clone()))
|
.map(|id| (id.clone(), parent.clone()))
|
||||||
.collect();
|
.collect();
|
||||||
let id = config::unique_slug(title, &taken);
|
let id = config::unique_slug(title, &taken);
|
||||||
@@ -1255,7 +1282,7 @@ pub struct Policy {
|
|||||||
|
|
||||||
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
|
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
|
||||||
let global = ctx.cfg().general.max_new_per_check;
|
let global = ctx.cfg().general.max_new_per_check;
|
||||||
Ok(merge_policy(&ctx.db.subscribers(id)?, feed_cfg, global))
|
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
|
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
|
||||||
|
|||||||
108
src/web.rs
108
src/web.rs
@@ -12,9 +12,7 @@ use axum::{
|
|||||||
},
|
},
|
||||||
routing::{delete, get, patch, post},
|
routing::{delete, get, patch, post},
|
||||||
};
|
};
|
||||||
use futures_util::StreamExt;
|
|
||||||
use serde::Deserialize;
|
use serde::Deserialize;
|
||||||
use tokio_stream::wrappers::BroadcastStream;
|
|
||||||
use tower::ServiceExt;
|
use tower::ServiceExt;
|
||||||
use tower_http::services::ServeFile;
|
use tower_http::services::ServeFile;
|
||||||
use serde::Serialize;
|
use serde::Serialize;
|
||||||
@@ -35,28 +33,6 @@ pub struct WebState {
|
|||||||
pub events: broadcast::Sender<Event>,
|
pub events: broadcast::Sender<Event>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A 32-hex-character shared secret, generated when config.toml has none.
|
|
||||||
///
|
|
||||||
/// ponytail: /dev/urandom rather than a CSPRNG crate -- 16 bytes, once, on a Unix-only
|
|
||||||
/// binary. Falls back to the clock only if urandom is somehow unreadable, which would be a
|
|
||||||
/// weak token, so that case is logged loudly.
|
|
||||||
pub fn generate_token() -> String {
|
|
||||||
use std::io::Read;
|
|
||||||
let mut bytes = [0u8; 16];
|
|
||||||
match std::fs::File::open("/dev/urandom").and_then(|mut f| f.read_exact(&mut bytes)) {
|
|
||||||
Ok(()) => {}
|
|
||||||
Err(e) => {
|
|
||||||
tracing::error!(error = %e, "could not read /dev/urandom; token is NOT secure");
|
|
||||||
let n = std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.map(|d| d.as_nanos() as u64)
|
|
||||||
.unwrap_or(0);
|
|
||||||
bytes[..8].copy_from_slice(&n.to_le_bytes());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
bytes.iter().map(|b| format!("{b:02x}")).collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn router(state: WebState) -> Router {
|
pub fn router(state: WebState) -> Router {
|
||||||
Router::new()
|
Router::new()
|
||||||
.route("/", get(index))
|
.route("/", get(index))
|
||||||
@@ -88,6 +64,7 @@ pub fn router(state: WebState) -> Router {
|
|||||||
// Signing in cannot require being signed in, so these sit outside the auth layer.
|
// Signing in cannot require being signed in, so these sit outside the auth layer.
|
||||||
.route("/login", get(login_page))
|
.route("/login", get(login_page))
|
||||||
.route("/api/login", post(login))
|
.route("/api/login", post(login))
|
||||||
|
.route("/icon.png", get(icon))
|
||||||
.layer(middleware::from_fn(access_log))
|
.layer(middleware::from_fn(access_log))
|
||||||
.with_state(state)
|
.with_state(state)
|
||||||
}
|
}
|
||||||
@@ -433,6 +410,15 @@ async fn login_page() -> Html<&'static str> {
|
|||||||
Html(include_str!("../web/login.html"))
|
Html(include_str!("../web/login.html"))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The 2004 icon, served once for both pages rather than inlined as base64 into each. The
|
||||||
|
/// sign-in page shows it, so it sits outside the auth layer with /login.
|
||||||
|
async fn icon() -> impl IntoResponse {
|
||||||
|
(
|
||||||
|
[(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")],
|
||||||
|
include_bytes!("../web/ipodderx-icon.png").as_slice(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
fn constant_time_eq(a: &str, b: &str) -> bool {
|
fn constant_time_eq(a: &str, b: &str) -> bool {
|
||||||
let (a, b) = (a.as_bytes(), b.as_bytes());
|
let (a, b) = (a.as_bytes(), b.as_bytes());
|
||||||
if a.len() != b.len() {
|
if a.len() != b.len() {
|
||||||
@@ -495,6 +481,9 @@ async fn feeds(
|
|||||||
let mut out = Vec::with_capacity(mine.len());
|
let mut out = Vec::with_capacity(mine.len());
|
||||||
for sub in &subs {
|
for sub in &subs {
|
||||||
let (id, feed) = (&sub.id, &sub.cfg);
|
let (id, feed) = (&sub.id, &sub.cfg);
|
||||||
|
// In a group, what you have not set on the feed comes from your settings on the group,
|
||||||
|
// the same fallback the scanner uses (`Db::subscribers`).
|
||||||
|
let up = feed.group.as_deref().and_then(|g| mine.get(g));
|
||||||
let Some(mine) = mine.get(id) else { continue };
|
let Some(mine) = mine.get(id) else { continue };
|
||||||
let s = state.ctx.db.feed_summary(id)?;
|
let s = state.ctx.db.feed_summary(id)?;
|
||||||
let st = state.ctx.db.http_state(id)?;
|
let st = state.ctx.db.http_state(id)?;
|
||||||
@@ -504,11 +493,22 @@ async fn feeds(
|
|||||||
title: s.title,
|
title: s.title,
|
||||||
image: s.image,
|
image: s.image,
|
||||||
folder: feed.folder.clone(),
|
folder: feed.folder.clone(),
|
||||||
keywords: mine.keywords.clone().unwrap_or_else(|| feed.keywords.clone()),
|
keywords: mine
|
||||||
allow_explicit: mine.allow_explicit.unwrap_or(feed.allow_explicit),
|
.keywords
|
||||||
auto_download: mine.auto_download.unwrap_or(feed.auto_download),
|
.clone()
|
||||||
|
.or_else(|| up.and_then(|u| u.keywords.clone()))
|
||||||
|
.unwrap_or_else(|| feed.keywords.clone()),
|
||||||
|
allow_explicit: mine
|
||||||
|
.allow_explicit
|
||||||
|
.or(up.and_then(|u| u.allow_explicit))
|
||||||
|
.unwrap_or(feed.allow_explicit),
|
||||||
|
auto_download: mine
|
||||||
|
.auto_download
|
||||||
|
.or(up.and_then(|u| u.auto_download))
|
||||||
|
.unwrap_or(feed.auto_download),
|
||||||
max_new_per_check: mine
|
max_new_per_check: mine
|
||||||
.max_new_per_check
|
.max_new_per_check
|
||||||
|
.or(up.and_then(|u| u.max_new_per_check))
|
||||||
.map(|n| n as usize)
|
.map(|n| n as usize)
|
||||||
.or(feed.max_new_per_check),
|
.or(feed.max_new_per_check),
|
||||||
group: feed.group.clone(),
|
group: feed.group.clone(),
|
||||||
@@ -803,15 +803,6 @@ mod tests {
|
|||||||
"another feed already has that URL"
|
"another feed already has that URL"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn generated_tokens_are_32_hex_chars_and_not_repeated() {
|
|
||||||
let a = generate_token();
|
|
||||||
let b = generate_token();
|
|
||||||
assert_eq!(a.len(), 32);
|
|
||||||
assert!(a.chars().all(|c| c.is_ascii_hexdigit()));
|
|
||||||
assert_ne!(a, b);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
@@ -893,6 +884,19 @@ struct NewFeed {
|
|||||||
folder: Option<String>,
|
folder: Option<String>,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
keywords: Vec<String>,
|
keywords: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
allow_explicit: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it
|
||||||
|
/// goes on your subscription, and before the first scan, which would otherwise skip every
|
||||||
|
/// explicit item.
|
||||||
|
fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
|
||||||
|
if allow {
|
||||||
|
let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() };
|
||||||
|
state.ctx.db.set_subscription(user_id, &sub)?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn add_feed(
|
async fn add_feed(
|
||||||
@@ -901,23 +905,28 @@ async fn add_feed(
|
|||||||
Json(body): Json<NewFeed>,
|
Json(body): Json<NewFeed>,
|
||||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||||
let mut cfg = (*state.ctx.cfg()).clone();
|
let mut cfg = (*state.ctx.cfg()).clone();
|
||||||
|
let url = crate::feed::expand_input(&body.url);
|
||||||
// Someone else may already have it. Then adding costs nothing: no second fetch, no
|
// Someone else may already have it. Then adding costs nothing: no second fetch, no
|
||||||
// second copy on disk, just another name against the same feed.
|
// second copy on disk, just another name against the same feed.
|
||||||
if let Some(existing) = crate::subscriptions(&state.ctx)?
|
if let Some(existing) = crate::subscriptions(&state.ctx)?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.find(|s| s.cfg.url == body.url)
|
.find(|s| crate::feed::same_feed(&s.cfg.url, &url))
|
||||||
{
|
{
|
||||||
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some();
|
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some();
|
||||||
state.ctx.db.subscribe(user.id, &existing.id)?;
|
state.ctx.db.subscribe(user.id, &existing.id)?;
|
||||||
|
if !already {
|
||||||
|
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit)?;
|
||||||
|
}
|
||||||
scan_soon(&state, Some(existing.id.clone())).await;
|
scan_soon(&state, Some(existing.id.clone())).await;
|
||||||
return Ok(Json(
|
return Ok(Json(
|
||||||
serde_json::json!({ "id": existing.id, "existing": already }),
|
serde_json::json!({ "id": existing.id, "existing": already }),
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
let id = crate::add_one(&state.ctx, &mut cfg, &body.url, body.folder, body.keywords).await?;
|
let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
|
||||||
cfg.save(&state.config_path)?;
|
cfg.save(&state.config_path)?;
|
||||||
state.ctx.reload_cfg(&state.config_path)?;
|
state.ctx.reload_cfg(&state.config_path)?;
|
||||||
state.ctx.db.subscribe(user.id, &id)?;
|
state.ctx.db.subscribe(user.id, &id)?;
|
||||||
|
explicit_on_add(&state, user.id, &id, body.allow_explicit)?;
|
||||||
scan_soon(&state, Some(id.clone())).await;
|
scan_soon(&state, Some(id.clone())).await;
|
||||||
Ok(Json(serde_json::json!({ "id": id, "existing": false })))
|
Ok(Json(serde_json::json!({ "id": id, "existing": false })))
|
||||||
}
|
}
|
||||||
@@ -1164,9 +1173,9 @@ async fn delete_file(
|
|||||||
let complaint = match (starred, unread) {
|
let complaint = match (starred, unread) {
|
||||||
(0, 0) => None,
|
(0, 0) => None,
|
||||||
(0, u) => Some(format!("{} subscribed to this feed {} not played it yet", people(u), if u == 1 { "has" } else { "have" })),
|
(0, u) => Some(format!("{} subscribed to this feed {} not played it yet", people(u), if u == 1 { "has" } else { "have" })),
|
||||||
(st, 0) => Some(format!("another {} starred it to keep", people(st))),
|
(st, 0) => Some(format!("another {} kept it", people(st))),
|
||||||
(st, u) => Some(format!(
|
(st, u) => Some(format!(
|
||||||
"another {} starred it to keep, and {} not played it yet",
|
"another {} kept it, and {} not played it yet",
|
||||||
people(st),
|
people(st),
|
||||||
if u == 1 { "one person has".to_string() } else { format!("{u} have") }
|
if u == 1 { "one person has".to_string() } else { format!("{u} have") }
|
||||||
)),
|
)),
|
||||||
@@ -1215,9 +1224,20 @@ async fn fetch_now(
|
|||||||
|
|
||||||
/// The same broadcast the socket clients read, as server-sent events.
|
/// The same broadcast the socket clients read, as server-sent events.
|
||||||
async fn events(State(state): State<WebState>) -> Sse<impl futures_util::Stream<Item = Result<SseEvent, std::convert::Infallible>>> {
|
async fn events(State(state): State<WebState>) -> Sse<impl futures_util::Stream<Item = Result<SseEvent, std::convert::Infallible>>> {
|
||||||
let stream = BroadcastStream::new(state.events.subscribe()).filter_map(|ev| async move {
|
// A client that falls behind skips what it missed rather than being cut off.
|
||||||
let ev = ev.ok()?;
|
let stream = futures_util::stream::unfold(state.events.subscribe(), |mut rx| async move {
|
||||||
Some(Ok(SseEvent::default().data(serde_json::to_string(&ev).ok()?)))
|
loop {
|
||||||
|
match rx.recv().await {
|
||||||
|
Ok(ev) => {
|
||||||
|
if let Ok(data) = serde_json::to_string(&ev) {
|
||||||
|
let ev = Ok::<_, std::convert::Infallible>(SseEvent::default().data(data));
|
||||||
|
return Some((ev, rx));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(broadcast::error::RecvError::Lagged(_)) => {}
|
||||||
|
Err(broadcast::error::RecvError::Closed) => return None,
|
||||||
|
}
|
||||||
|
}
|
||||||
});
|
});
|
||||||
Sse::new(stream).keep_alive(axum::response::sse::KeepAlive::default())
|
Sse::new(stream).keep_alive(axum::response::sse::KeepAlive::default())
|
||||||
}
|
}
|
||||||
@@ -1428,8 +1448,6 @@ async fn patch_settings(
|
|||||||
)));
|
)));
|
||||||
}
|
}
|
||||||
cfg.general.schedule = sched;
|
cfg.general.schedule = sched;
|
||||||
// The legacy key would otherwise keep shadowing intent in the file.
|
|
||||||
cfg.general.interval_mins = None;
|
|
||||||
}
|
}
|
||||||
if let Some(v) = body.max_new_per_check {
|
if let Some(v) = body.max_new_per_check {
|
||||||
cfg.general.max_new_per_check = v;
|
cfg.general.max_new_per_check = v;
|
||||||
|
|||||||
@@ -142,7 +142,7 @@ test('the filter tabs change what is listed', async ({ page }) => {
|
|||||||
await page.locator('.tabs button', { hasText: 'Unread' }).first().click();
|
await page.locator('.tabs button', { hasText: 'Unread' }).first().click();
|
||||||
expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all);
|
expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all);
|
||||||
|
|
||||||
await page.locator('.tabs button', { hasText: 'Flagged' }).first().click();
|
await page.locator('.tabs button', { hasText: 'Kept' }).first().click();
|
||||||
await expect(page.locator('#count')).toContainText('0 items');
|
await expect(page.locator('#count')).toContainText('0 items');
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -187,9 +187,11 @@ test('an OPML subscription is a collapsible folder', async ({ page }) => {
|
|||||||
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
||||||
}));
|
}));
|
||||||
|
|
||||||
// Every row reserves the chevron slot for alignment; only a folder's is clickable.
|
// Only a folder has a triangle, and it is a button that says whether the folder is open.
|
||||||
const chev = page.locator('.feed.group .chev');
|
const chev = page.locator('.feed.group .chev');
|
||||||
await expect(chev).toBeVisible({ timeout: 20_000 });
|
await expect(chev).toBeVisible({ timeout: 20_000 });
|
||||||
|
await expect(page.locator('.feed:not(.group) .chev')).toHaveCount(0);
|
||||||
|
await expect(chev).toHaveAttribute('aria-expanded', 'false');
|
||||||
|
|
||||||
// Closed by default: the children are not listed until the folder is opened.
|
// Closed by default: the children are not listed until the folder is opened.
|
||||||
const before = await page.locator('.feed').count();
|
const before = await page.locator('.feed').count();
|
||||||
@@ -346,6 +348,10 @@ test('a second person has their own feeds and their own read state', async ({ br
|
|||||||
const ctx = await browser.newContext();
|
const ctx = await browser.newContext();
|
||||||
const page = await ctx.newPage();
|
const page = await ctx.newPage();
|
||||||
await page.goto('/login');
|
await page.goto('/login');
|
||||||
|
// The sign-in page shows the icon, so it has to load before anyone has signed in.
|
||||||
|
const icon = await page.request.get('/icon.png');
|
||||||
|
expect(icon.status()).toBe(200);
|
||||||
|
expect(icon.headers()['content-type']).toBe('image/png');
|
||||||
await page.locator('#name').fill('sam');
|
await page.locator('#name').fill('sam');
|
||||||
await page.locator('#pw').fill('sampassword');
|
await page.locator('#pw').fill('sampassword');
|
||||||
await page.locator('button[type=submit]').click();
|
await page.locator('button[type=submit]').click();
|
||||||
|
|||||||
265
web/index.html
265
web/index.html
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user