Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9aae3097e7 | |||
| 2ff2074755 | |||
| 1698cf8d1e | |||
| b94a74ef15 | |||
| 9a8a3c696f | |||
| bedf64e645 | |||
| 586d2c07a1 | |||
| 2ba83c3aed | |||
| 1352f0d54d | |||
| a958f7cb37 | |||
| 457a58dcc5 | |||
| 2af57065c6 | |||
| 564b011c7a | |||
| 9eb7aadced | |||
| c6bceaef37 | |||
| dc63d6acaf | |||
| 8937f35f00 | |||
| 0990f2a90d | |||
| 0cc002cdfa | |||
| 2c9e899762 | |||
| 2e416f96cf | |||
| 9269aa99f7 |
101
CHANGELOG.md
101
CHANGELOG.md
@@ -10,6 +10,102 @@ The long form, with what was wrong before and how it was found, is in
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.5.2] - 2026-09-12
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Settings → Users and `ipx user list` show when each account was added and when it last signed
|
||||||
|
in, to the hour.
|
||||||
|
- `ipx user rename <name> <new name>` renames an account and keeps its feeds, read state and admin
|
||||||
|
rights. An account made before the proxy was set up can take the name the proxy signs it in as.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Directory and Popular list the feeds inside an OPML one by one, and no longer the OPML itself,
|
||||||
|
so you can subscribe to just the shows you want.
|
||||||
|
- The database no longer records when subscriptions and sign-in sessions were created. Nothing
|
||||||
|
ever read it, and an existing database drops the columns on its next start.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Show notes that the podcast's host cut off in the middle of a tag no longer open with a scrap of
|
||||||
|
HTML: the item's other copy of its notes is used instead, from the next time the feed changes.
|
||||||
|
Daily Meditation Podcast had 57.
|
||||||
|
- Docker no longer shows ipodderx as starting, or calls it unhealthy, while it scans or downloads:
|
||||||
|
`ipx status` answers at once instead of waiting for the job in progress to finish.
|
||||||
|
- Signing out after signing in through Cloudflare Access no longer lands on ipodderx's own password
|
||||||
|
page. With the new `sign_out_url` set, Sign out ends the Access session, and the password page
|
||||||
|
sends anyone the proxy signs in straight to their feeds.
|
||||||
|
- The sign-in guide, `docs/sso.md`, describes the setup ipodderx.sdf1.net really runs: Authentik as
|
||||||
|
Cloudflare Access's identity provider, and how to find the address ipx has to trust. It had never
|
||||||
|
been checked against a real setup, and pointed at the wrong address.
|
||||||
|
|
||||||
|
## [0.5.1] - 2026-09-12
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- The triangle that opens an OPML or Patreon folder was cramped against the folder's art. It has
|
||||||
|
more room now, and a wider target to click.
|
||||||
|
|
||||||
|
## [0.5.0] - 2026-09-12
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- A Patreon token pasted into Add feed, or a creator's RSS link without `&show=`, becomes a folder
|
||||||
|
of that creator's shows, kept in step on every scan like a subscribed OPML. A creator with only
|
||||||
|
one show stays a plain feed. One already added as a single long feed is split into its shows on
|
||||||
|
its next scan, keeping its files and what you had read.
|
||||||
|
- Add feed has an "Allow items marked explicit" box, so a new feed's first scan no longer skips
|
||||||
|
every explicit item.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Unread counts, unread dots and download progress are amber, the colour of the icon's EQ bars.
|
||||||
|
Blue is kept for the primary action and links, so a count no longer looks like a button.
|
||||||
|
- What is playing is marked by small EQ bars, in its row and in the player. They move only while it
|
||||||
|
plays.
|
||||||
|
- A folder in the sidebar shows its first four shows' art as a mosaic, and its shows sit under its
|
||||||
|
title. Only folders have a triangle, so every feed lines up with Directory and Popular above.
|
||||||
|
- Feeds without art get initials in a colour of their own, instead of all the same grey.
|
||||||
|
- The Flagged tab is called Kept, as the Keep button and Settings already said.
|
||||||
|
- A feed's header is one short line; when it checks next is in its tooltip.
|
||||||
|
- The item list takes more of the window, and the Files pane shows only when the item has files.
|
||||||
|
- Column headings and tags are in sentence case, and fewer things are bold.
|
||||||
|
- Unsubscribe is a round button beside the feed's other actions.
|
||||||
|
- Export and Import in Settings say what they do.
|
||||||
|
- The sign-in page shows the original icon large.
|
||||||
|
- Nothing animates when your system asks for reduced motion.
|
||||||
|
- The pages are about 90 KB smaller: the icon is served once instead of written into each.
|
||||||
|
- A web token generated for a new install is 64 characters instead of 32.
|
||||||
|
- The README is a short overview of what ipx does and how to run it, and points into `docs/` for
|
||||||
|
the rest. It still described the layout from before 0.4.0.
|
||||||
|
|
||||||
|
### Removed
|
||||||
|
|
||||||
|
- The systemd units in `contrib/`. Run ipx with Docker, or point a unit of your own at
|
||||||
|
`ipx daemon`.
|
||||||
|
- Upgrading from before 0.3.0 directly: what was read, kept or part-played before accounts is no
|
||||||
|
longer carried over to the admin, and OPML feeds that old versions wrote into `config.toml` are
|
||||||
|
no longer moved out of it. Upgrade through 0.4.0 first.
|
||||||
|
- `interval_mins` in `config.toml` is ignored; use `schedule`.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- The feed list works from the keyboard: Tab reaches every feed, Enter opens it, and Right and Left
|
||||||
|
open and close a folder. Every button shows where the focus is, including in the toolbar, which
|
||||||
|
used to clip the ring.
|
||||||
|
- "1 items" reads "1 item".
|
||||||
|
- A selected feed without art no longer loses its initials tile in Dark and Light.
|
||||||
|
- The player shows the feed's initials when there is no art, not the episode's.
|
||||||
|
- Turning on Allow explicit, or changing keywords or auto-download, brings back what those settings
|
||||||
|
had skipped on the feed's next scan. Before, an item was judged once, when first seen, and a
|
||||||
|
skipped one stayed skipped whatever you changed.
|
||||||
|
- Feeds inside an OPML or a Patreon creator follow your settings on the folder unless you set their
|
||||||
|
own, as the folder's settings dialog said they did. Before, the folder's settings reached nothing
|
||||||
|
inside it.
|
||||||
|
- A new feed no longer takes the name of one you removed earlier and shows that feed's old items.
|
||||||
|
Re-adding the same feed still gets its old name, and its history, back.
|
||||||
|
|
||||||
## [0.4.0] - 2026-09-11
|
## [0.4.0] - 2026-09-11
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
@@ -199,7 +295,10 @@ The long form, with what was wrong before and how it was found, is in
|
|||||||
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
|
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
|
||||||
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
|
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
|
||||||
|
|
||||||
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...main
|
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.2...main
|
||||||
|
[0.5.2]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.1...v0.5.2
|
||||||
|
[0.5.1]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.0...v0.5.1
|
||||||
|
[0.5.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...v0.5.0
|
||||||
[0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0
|
[0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0
|
||||||
[0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0
|
[0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0
|
||||||
[0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0
|
[0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0
|
||||||
|
|||||||
21
CLAUDE.md
21
CLAUDE.md
@@ -17,6 +17,7 @@ Arcane project `content`: `/mnt/fast/arcane/projects/content/compose.yaml`. That
|
|||||||
| Database | `/mnt/user/ipodderx/state.db` | `/data/state.db` |
|
| Database | `/mnt/user/ipodderx/state.db` | `/data/state.db` |
|
||||||
| Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` |
|
| Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` |
|
||||||
| Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` |
|
| Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` |
|
||||||
|
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | trusts `Cf-Access-Authenticated-User-Email` from `192.168.16.1`, the `content_default` gateway |
|
||||||
|
|
||||||
Deploying a change is: build and push the image, then pull it and recreate the container.
|
Deploying a change is: build and push the image, then pull it and recreate the container.
|
||||||
|
|
||||||
@@ -45,7 +46,10 @@ docker tag mirror.gcr.io/library/rust:1-slim-bookworm rust:1-slim-bookworm
|
|||||||
Run those again now and then, or the local copies go stale.
|
Run those again now and then, or the local copies go stale.
|
||||||
|
|
||||||
The healthcheck runs `ipx status` against the control socket, so `(healthy)` in `docker ps` means
|
The healthcheck runs `ipx status` against the control socket, so `(healthy)` in `docker ps` means
|
||||||
the worker is alive, not just the web port. The container restarts on its own after a reboot.
|
the daemon answers there and can read its database, not just that the web port is up. The socket
|
||||||
|
answers `status` itself instead of queuing it behind the worker's current job, so a long scan or
|
||||||
|
download does not fail the check; it also means a worker stuck on one job would still pass. The
|
||||||
|
container restarts on its own after a reboot.
|
||||||
|
|
||||||
Before the container, ipx ran by hand in code-server, with its files in `/config/.config/ipx/` and
|
Before the container, ipx ran by hand in code-server, with its files in `/config/.config/ipx/` and
|
||||||
`/config/.local/share/ipx/`. Those are still there and the container does not read them. If you run
|
`/config/.local/share/ipx/`. Those are still there and the container does not read them. If you run
|
||||||
@@ -99,9 +103,9 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
|
|||||||
|
|
||||||
* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
|
* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
|
||||||
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
|
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
|
||||||
* **`entries.read`, `entries.flagged` and `entries.position` are dead columns.** Read state lives in
|
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
|
||||||
`entry_state` per user. Two bugs have already come from queries still reading the old ones
|
and `position` columns from before accounts; two bugs came from queries still reading them
|
||||||
(retention, and the entry pruner) — grep before adding a third.
|
(retention, and the entry pruner), and `migrate()` now drops them.
|
||||||
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
|
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
|
||||||
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
|
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
|
||||||
derived and never written to config.
|
derived and never written to config.
|
||||||
@@ -113,8 +117,13 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
|
|||||||
watch the shutdown channel itself; the daemon ignored SIGTERM for exactly this reason.
|
watch the shutdown channel itself; the daemon ignored SIGTERM for exactly this reason.
|
||||||
* Only one daemon per socket. Removing the socket file defeats the guard and you get two daemons
|
* Only one daemon per socket. Removing the socket file defeats the guard and you get two daemons
|
||||||
fighting over the database, with the stale one still holding the port.
|
fighting over the database, with the stale one still holding the port.
|
||||||
* `/api/settings` answering `200` does **not** mean the worker is alive — it is a different task.
|
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
|
||||||
Probe the control socket (`ipx status`) to check that.
|
different task. `ipx status` checks the control socket and the database; to see the worker
|
||||||
|
getting through its jobs, watch for `scan complete` in the log.
|
||||||
|
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's
|
||||||
|
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on
|
||||||
|
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait
|
||||||
|
for `daemon started` in the log. Copy `state.db` aside before deploying one.
|
||||||
|
|
||||||
## House style
|
## House style
|
||||||
|
|
||||||
|
|||||||
34
Cargo.lock
generated
34
Cargo.lock
generated
@@ -436,17 +436,6 @@ dependencies = [
|
|||||||
"shlex",
|
"shlex",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "cfb"
|
|
||||||
version = "0.14.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "a347dcabdae9c31b0825fd6a8bed285ec9c2acb89c47827126d52fa4f59cece3"
|
|
||||||
dependencies = [
|
|
||||||
"fnv",
|
|
||||||
"uuid",
|
|
||||||
"web-time",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "cfg-if"
|
name = "cfg-if"
|
||||||
version = "1.0.4"
|
version = "1.0.4"
|
||||||
@@ -867,15 +856,6 @@ dependencies = [
|
|||||||
"dirs-sys",
|
"dirs-sys",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "dirs"
|
|
||||||
version = "7.0.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "8d57d423b3c82e89b9a24ca3091fee61f456a26edbd28d26c65906f4bc1dcd8f"
|
|
||||||
dependencies = [
|
|
||||||
"dirs-sys",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "dirs-sys"
|
name = "dirs-sys"
|
||||||
version = "0.5.0"
|
version = "0.5.0"
|
||||||
@@ -1608,15 +1588,6 @@ dependencies = [
|
|||||||
"serde_core",
|
"serde_core",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "infer"
|
|
||||||
version = "0.22.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "f4200d433cbd5178df7797c9c2e75b348b728e39631cf14520d1e2fc424201f4"
|
|
||||||
dependencies = [
|
|
||||||
"cfb",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "intervaltree"
|
name = "intervaltree"
|
||||||
version = "0.2.7"
|
version = "0.2.7"
|
||||||
@@ -1634,7 +1605,7 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "ipx"
|
name = "ipx"
|
||||||
version = "0.4.0"
|
version = "0.5.2"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"ammonia",
|
"ammonia",
|
||||||
"anyhow",
|
"anyhow",
|
||||||
@@ -1643,9 +1614,7 @@ dependencies = [
|
|||||||
"axum",
|
"axum",
|
||||||
"chrono",
|
"chrono",
|
||||||
"clap",
|
"clap",
|
||||||
"dirs",
|
|
||||||
"futures-util",
|
"futures-util",
|
||||||
"infer",
|
|
||||||
"librqbit",
|
"librqbit",
|
||||||
"opml",
|
"opml",
|
||||||
"percent-encoding",
|
"percent-encoding",
|
||||||
@@ -1656,7 +1625,6 @@ dependencies = [
|
|||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
"tokio",
|
"tokio",
|
||||||
"tokio-stream",
|
|
||||||
"toml",
|
"toml",
|
||||||
"tower",
|
"tower",
|
||||||
"tower-http 0.7.1",
|
"tower-http 0.7.1",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "ipx"
|
name = "ipx"
|
||||||
version = "0.4.0"
|
version = "0.5.2"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
@@ -11,9 +11,7 @@ atom_syndication = "0.12.10"
|
|||||||
axum = "0.8.9"
|
axum = "0.8.9"
|
||||||
chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] }
|
chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] }
|
||||||
clap = { version = "4.6.6", features = ["derive"] }
|
clap = { version = "4.6.6", features = ["derive"] }
|
||||||
dirs = "7.0.0"
|
|
||||||
futures-util = { version = "0.3.34", default-features = false, features = ["std"] }
|
futures-util = { version = "0.3.34", default-features = false, features = ["std"] }
|
||||||
infer = "0.22.0"
|
|
||||||
librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] }
|
librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] }
|
||||||
opml = "1.1.6"
|
opml = "1.1.6"
|
||||||
percent-encoding = "2.3.2"
|
percent-encoding = "2.3.2"
|
||||||
@@ -24,7 +22,6 @@ rusqlite = { version = "0.40.2", features = ["bundled"] }
|
|||||||
serde = { version = "1.0.229", features = ["derive"] }
|
serde = { version = "1.0.229", features = ["derive"] }
|
||||||
serde_json = "1.0.151"
|
serde_json = "1.0.151"
|
||||||
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
|
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
|
||||||
tokio-stream = { version = "0.1.19", features = ["sync"] }
|
|
||||||
toml = "1.1.5"
|
toml = "1.1.5"
|
||||||
tower = { version = "0.5.3", features = ["util"] }
|
tower = { version = "0.5.3", features = ["util"] }
|
||||||
tower-http = { version = "0.7.1", features = ["fs"] }
|
tower-http = { version = "0.7.1", features = ["fs"] }
|
||||||
|
|||||||
144
README.md
144
README.md
@@ -1,123 +1,83 @@
|
|||||||
# ipodderx-rs
|
# ipodderx-rs
|
||||||
|
|
||||||
A headless podcatcher: scans RSS/Atom feeds, downloads enclosures (HTTP and BitTorrent), files them
|
A self-hosted podcatcher for a household. It checks your feeds, downloads the episodes, and serves
|
||||||
into per-feed folders, and reaps old files to stay under a disk quota. Runs as a one-shot CLI or as
|
a web UI modelled on the 2004 Mac app **iPodderX**, for any number of people sharing one copy of
|
||||||
a daemon with a web UI, serving any number of people from one copy of the data.
|
the files. One Rust binary, `ipx`, is both the daemon and the command line.
|
||||||
|
|
||||||
## Lineage
|
It is a rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python engine
|
||||||
|
behind iPodderX (2004-2008, Ray Slakinski & August Trometer).
|
||||||
|
|
||||||
A modern Rust rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python 2
|
## What it does
|
||||||
engine behind **iPodderX** (2004-2008, Ray Slakinski & August Trometer), open-sourced under the MIT
|
|
||||||
License in 2010.
|
|
||||||
|
|
||||||
What carries over: the feed scan and TTL handling, GUID/URL dedupe, per-feed and per-date download
|
- **The web UI.** It has a toolbar, and a feed list that opens with Directory, Popular and All
|
||||||
folders, keyword filters, the explicit-content filter, torrent enclosures, and "SmartSpace" -- the
|
Subscriptions. Items sit in a sortable table with a Files pane, and there is a player bar. It
|
||||||
oldest-first disk quota reaper.
|
comes in Dark, Light and Classic themes, and works on a phone.
|
||||||
|
- **Several people, one copy.** Each person has their own subscriptions and their own read, kept
|
||||||
|
and playback state. There is one file on disk per episode, however many people want it. People
|
||||||
|
sign in with a password or through a proxy (Cloudflare Zero Trust or Authentik), and admins
|
||||||
|
manage accounts and settings.
|
||||||
|
- **Scanning.** Feeds are checked on a schedule, globally or per feed, and a feed's own TTL is
|
||||||
|
honoured. Keyword, explicit-content and media-type filters decide what is downloaded, with a cap
|
||||||
|
on new downloads per scan.
|
||||||
|
- **Downloads.** Files come over HTTP or BitTorrent and are filed into a folder per feed.
|
||||||
|
Retention deletes the oldest files to stay under a disk quota or an age limit, and never touches
|
||||||
|
an item someone has kept.
|
||||||
|
- **OPML.** You can import and export your own subscriptions. You can also subscribe to an OPML
|
||||||
|
URL, which keeps a whole list in step as a folder.
|
||||||
|
|
||||||
What does not: iTunes and iPhoto export via AppleScript, text-to-speech enclosures, the Windows
|
## Run it
|
||||||
WMP/COM paths, XML plists and Python pickles for state, the `directory.iPodderX.com` survey ping,
|
|
||||||
3DES-encrypted preferences, and the `printMSG` stdout protocol -- replaced by a JSON-lines socket.
|
|
||||||
|
|
||||||
## Quick start
|
With Docker:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker build -t ipodderx .
|
||||||
|
docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
`docker-compose.yml` is set up for the author's own server. Point its `image` and its three volumes
|
||||||
|
(`/config`, `/data` and `/downloads`) at yours first. The UI is on port 8099. BitTorrent uses 6881
|
||||||
|
over TCP and UDP. Files are written as `PUID`/`PGID`, 99:100 by default.
|
||||||
|
|
||||||
|
From source:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cargo build --release
|
cargo build --release
|
||||||
install -m755 target/release/ipx ~/.cargo/bin/
|
./target/release/ipx daemon
|
||||||
|
|
||||||
ipx add https://atp.fm/rss # subscribe
|
|
||||||
ipx fetch # scan and download
|
|
||||||
ipx daemon # scheduler, control socket and web UI
|
|
||||||
```
|
```
|
||||||
|
|
||||||
On first start with `[web] enabled = true` the daemon mints a token, writes it to config.toml and
|
The first start creates **admin / ipodderx**. Sign in at `/login`, then change it:
|
||||||
prints the URL to open. A database with no accounts starts with **admin / ipodderx** at `/login` --
|
|
||||||
change it with `echo -n '<password>' | ipx user passwd admin`.
|
```sh
|
||||||
|
echo -n 'a good password' | ipx user passwd admin
|
||||||
|
```
|
||||||
|
|
||||||
|
The UI is plain HTTP, so put TLS in front of it if it is reachable from outside your network.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
| [docs/configuration.md](docs/configuration.md) | Every config key, paths, environment variables |
|
| [docs/configuration.md](docs/configuration.md) | Every config key, path and environment variable |
|
||||||
| [docs/cli.md](docs/cli.md) | Every command, including `ipx user` |
|
| [docs/cli.md](docs/cli.md) | Every command, including `ipx user` |
|
||||||
| [docs/users.md](docs/users.md) | Accounts, and what several people share |
|
| [docs/users.md](docs/users.md) | Accounts, and what several people share |
|
||||||
| [docs/sso.md](docs/sso.md) | Cloudflare Zero Trust or Authentik in front |
|
| [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik |
|
||||||
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, socket, HTTP API |
|
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API |
|
||||||
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
|
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
|
||||||
| [docs/history.md](docs/history.md) | How it was built: the long form, with what was wrong and why |
|
| [docs/history.md](docs/history.md) | How it was built, with what was wrong and why |
|
||||||
| [CLAUDE.md](CLAUDE.md) | Notes for anyone (or anything) working on the code |
|
| [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed |
|
||||||
|
|
||||||
## The web UI
|
|
||||||
|
|
||||||
`ipx daemon` serves it in the same process, so it reads SQLite and the event bus directly.
|
|
||||||
|
|
||||||
Feeds down the side; the selected feed's items across the top; the selected item's text and its
|
|
||||||
enclosures below, which is where you play, download or delete them. The divider drags and its
|
|
||||||
position is remembered. Playback serves Range requests, so seeking works. An OPML subscription is a
|
|
||||||
collapsible folder whose page lists the feeds inside it.
|
|
||||||
|
|
||||||
An item may carry several enclosures; all of them appear below, and anything that is not audio or
|
|
||||||
video gets a View link rather than a player -- the publisher's copy until it is downloaded, the
|
|
||||||
local one after. Opening an item marks it read. Show notes are untrusted feed HTML, sanitized with
|
|
||||||
`ammonia` server-side before they reach the page.
|
|
||||||
|
|
||||||
The **Log** button shows the running daemon live in four tabs: *Daemon I/O* is the control protocol
|
|
||||||
itself, every command in and event out; *Scans* is feed and download activity; *HTTP* is web
|
|
||||||
requests; *All* is everything, with level and text filters and a copy button. It reads a ring buffer
|
|
||||||
held in the process, not a file, so it works the same under Docker.
|
|
||||||
|
|
||||||
It is plain HTTP. On a LAN bind the token and everything else cross the network in the clear, and a
|
|
||||||
feed URL can itself carry a credential. Put TLS in front of it if that matters.
|
|
||||||
|
|
||||||
## OPML
|
|
||||||
|
|
||||||
**Importing and exporting** a file copies subscriptions in or out once: `ipx import subs.opml`,
|
|
||||||
`ipx export subs.opml`, or Settings → Subscriptions in the UI.
|
|
||||||
|
|
||||||
**Subscribing to an OPML URL** is a live subscription, as iPodderX had. Add the OPML's URL like any
|
|
||||||
other feed; every scan re-reads it and keeps your list in step. The feeds inside are not written to
|
|
||||||
config.toml -- the OPML is the source of truth, so they are re-derived each scan and held in the
|
|
||||||
database. They show as a folder, download into one nested folder, and inherit the subscription's
|
|
||||||
settings until you change one, which gives it its own entry.
|
|
||||||
|
|
||||||
When a feed drops out of the OPML upstream, it is unsubscribed and removed -- unless it has
|
|
||||||
downloads, in which case it is kept and flagged in the UI as no longer listed. A downloaded file is
|
|
||||||
never left behind with nothing explaining where it came from.
|
|
||||||
|
|
||||||
## Docker
|
|
||||||
|
|
||||||
```sh
|
|
||||||
docker buildx build --tag 192.168.1.130:5000/ipodderx:latest . --push
|
|
||||||
docker compose pull ipodderx && docker compose up -d ipodderx
|
|
||||||
docker compose logs -f ipodderx # the first start prints the default admin password
|
|
||||||
```
|
|
||||||
|
|
||||||
`docker-compose.yml` runs the image from the registry above rather than building it, so build and
|
|
||||||
push first; change the tag in both places to use another registry. It mounts `/config` (config.toml),
|
|
||||||
`/data` (state.db) and `/downloads` from this install's host paths, which you will want to change for
|
|
||||||
yours. It publishes 8099 for the
|
|
||||||
UI and 6881 (TCP **and** UDP -- DHT needs the UDP side), and sets `PUID`/`PGID` to `99:100` so files
|
|
||||||
land owned the way Unraid shares expect. The healthcheck runs `ipx status` through the control
|
|
||||||
socket, so it catches a daemon that is alive but wedged rather than merely one that has died.
|
|
||||||
|
|
||||||
## Running it as a service
|
|
||||||
|
|
||||||
`contrib/` has a systemd user unit for the daemon, and a timer plus one-shot service if you would
|
|
||||||
rather run periodic scans with no daemon -- in which case there is no socket for a UI to attach to.
|
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state
|
cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state
|
||||||
node tests/page-smoke.js # the page script loads without throwing
|
node tests/page-smoke.js # the page script loads without throwing
|
||||||
npx playwright test # a real browser against a real daemon
|
npx playwright test # a real browser against a real daemon on fixture feeds
|
||||||
```
|
```
|
||||||
|
|
||||||
`npm install` gets the test runner; the browser comes from
|
`npm install` gets the test runner, and `npx playwright install --with-deps chromium` gets the
|
||||||
`npx playwright install --with-deps chromium` (in `install.sh`).
|
browser.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
MIT. See [LICENSE](LICENSE).
|
MIT, see [LICENSE](LICENSE). The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by
|
||||||
|
@fontawesome, under [CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG.
|
||||||
The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by @fontawesome, under
|
|
||||||
[CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG in `web/index.html`.
|
|
||||||
|
|||||||
27
TODO.md
Normal file
27
TODO.md
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
# To do
|
||||||
|
|
||||||
|
## Trim the state database
|
||||||
|
|
||||||
|
From an audit of the database layer and a read-only copy of production on 2026-09-12. The data
|
||||||
|
itself was clean: no leftover tables or indexes, 47 free pages, one stray `entry_state` row.
|
||||||
|
Check each against the code before cutting it.
|
||||||
|
|
||||||
|
- [x] **`migrate()`'s add list.** All eight columns arrived in 0.2.0, and 0.5.0 only supports
|
||||||
|
upgrades from 0.3.0 on. Drop the list and its loop; keep the `retired` drop loop, which a
|
||||||
|
database coming from 0.4.0 still needs. (`src/db.rs`)
|
||||||
|
- [x] **`Db::subscribed_feed_ids`.** No callers; its doc says the scanner walks it, and it does not.
|
||||||
|
(`src/db.rs`)
|
||||||
|
- [x] **`Db::subscriber_count`.** One caller, which only asks whether it is above zero:
|
||||||
|
`subscriber_counts()?.contains_key(&id)`. (`src/db.rs`, `src/web.rs`)
|
||||||
|
- [x] **The `created` columns** on `users`, `subscriptions` and `sessions`. Written on every insert,
|
||||||
|
never read. Add them to `retired` and drop them from the inserts. (`src/db.rs`)
|
||||||
|
- [x] **`Managed.orphaned`.** Selected by `managed_feeds()` on every call and never read;
|
||||||
|
`FeedSummary.orphaned` is what the UI uses. (`src/db.rs`)
|
||||||
|
|
||||||
|
## Popular
|
||||||
|
|
||||||
|
- [x] **Popular lists feeds the way Directory does**: the feeds inside an OPML or a Patreon
|
||||||
|
creator, never the collection itself. (`src/web.rs`, `web/index.html`)
|
||||||
|
|
||||||
|
After these: `cargo test`, `node tests/page-smoke.js`, `npx playwright test`. Copy `state.db`
|
||||||
|
aside before deploying: `migrate()` drops columns on the first start.
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=ipx feed scan (one shot)
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=oneshot
|
|
||||||
ExecStart=%h/.cargo/bin/ipx fetch
|
|
||||||
Environment=IPX_LOG=ipx=info
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
# User unit: install to ~/.config/systemd/user/ipx.service, then
|
|
||||||
# systemctl --user enable --now ipx
|
|
||||||
# The socket lands in $XDG_RUNTIME_DIR/ipx.sock by default, so a UI running as the
|
|
||||||
# same user can attach without extra configuration.
|
|
||||||
[Unit]
|
|
||||||
Description=ipx podcatcher
|
|
||||||
After=network-online.target
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=simple
|
|
||||||
ExecStart=%h/.cargo/bin/ipx daemon
|
|
||||||
Restart=on-failure
|
|
||||||
RestartSec=30
|
|
||||||
Environment=IPX_LOG=ipx=info
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=default.target
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
# Alternative to the daemon: a periodic one-shot scan, closer to how the original
|
|
||||||
# iPodderX agent was driven. Use this OR ipx.service, not both -- with no daemon
|
|
||||||
# running there is no socket, so a UI cannot attach.
|
|
||||||
#
|
|
||||||
# Install ipx-scan.service and ipx.timer to ~/.config/systemd/user/, then
|
|
||||||
# systemctl --user enable --now ipx.timer
|
|
||||||
[Unit]
|
|
||||||
Description=Periodic ipx feed scan
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
OnBootSec=5min
|
|
||||||
OnUnitActiveSec=1h
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
@@ -28,12 +28,15 @@ The page is compiled in, so **editing `web/index.html` needs a rebuild**.
|
|||||||
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
|
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
|
||||||
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
|
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
|
||||||
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
|
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
|
||||||
re-derived into the database each scan, never written to config.toml.
|
re-derived into the database each scan, never written to config.toml. A Patreon creator link
|
||||||
|
(a token, no `show=`) with more than one show is treated the same way, before any fetch: its
|
||||||
|
shows come from Patreon's web API and each becomes a derived feed.
|
||||||
4. Record entries. A changed title or description flips the item back to unread.
|
4. Record entries. A changed title or description flips the item back to unread.
|
||||||
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
|
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
|
||||||
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
|
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
|
||||||
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
|
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
|
||||||
a reason.
|
a reason. What a filter skipped is judged again every scan, so a change of settings brings it
|
||||||
|
back. A feed in a group takes your settings on the group for anything you have not set on it.
|
||||||
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
|
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
|
||||||
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
|
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
|
||||||
or an error page — and is deleted.
|
or an error page — and is deleted.
|
||||||
@@ -47,21 +50,23 @@ entries feed_id, guid, title, link, published, description, first_seen,
|
|||||||
image, duration, episode, season PK (feed_id, guid)
|
image, duration, episode, season PK (feed_id, guid)
|
||||||
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,
|
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,
|
||||||
bytes_done, downloaded_at, last_error
|
bytes_done, downloaded_at, last_error
|
||||||
users id, name, pass_hash, is_admin, created
|
users id, name, pass_hash, is_admin, created, last_login
|
||||||
sessions token, user_id, created, seen
|
sessions token, user_id, seen
|
||||||
subscriptions user_id, feed_id, keywords, auto_download, allow_explicit,
|
subscriptions user_id, feed_id, keywords, auto_download, allow_explicit,
|
||||||
max_new_per_check, created PK (user_id, feed_id)
|
max_new_per_check PK (user_id, feed_id)
|
||||||
entry_state user_id, feed_id, guid, read, flagged, position
|
entry_state user_id, feed_id, guid, read, flagged, position
|
||||||
PK (user_id, feed_id, guid)
|
PK (user_id, feed_id, guid)
|
||||||
```
|
```
|
||||||
|
|
||||||
`entries` still has `read`, `flagged` and `position` columns from before accounts existed. They are
|
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
|
||||||
**dead** — the migration copied them into `entry_state` and nothing reads them now. Anything found
|
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
|
||||||
querying them is a bug; two were.
|
older database.
|
||||||
|
|
||||||
Schema changes: add the table or column to `SCHEMA`, and for a column also to the list in
|
Schema changes: add the table or column to `SCHEMA`. `CREATE TABLE IF NOT EXISTS` leaves a table
|
||||||
`migrate()`, which does `PRAGMA table_info` then `ALTER TABLE ADD COLUMN`. `Db::memory()` runs the
|
that already exists alone, so a new column on one also goes in `migrate()`'s `wanted` list, and a
|
||||||
same path as `Db::open`, so a migration-only column cannot pass tests while missing in production.
|
retired one in its `retired` list; both are checked with `PRAGMA table_info`. Columns from before
|
||||||
|
0.3.0, the oldest version an upgrade may start from, need no entry. `Db::memory()` runs the same
|
||||||
|
path as `Db::open`, so a migration cannot pass the tests while missing in production.
|
||||||
|
|
||||||
## Control socket
|
## Control socket
|
||||||
|
|
||||||
@@ -82,7 +87,9 @@ printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/
|
|||||||
**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
|
**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
|
||||||
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
|
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
|
||||||
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
|
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
|
||||||
there.
|
there. Commands run one at a time, in the order they arrive, except `status`: the socket answers it
|
||||||
|
straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and
|
||||||
|
answers only the client that asked, since `status` would end any other client's session.
|
||||||
|
|
||||||
Progress carries the enclosure id, without which a UI cannot tell one download from another and
|
Progress carries the enclosure id, without which a UI cannot tell one download from another and
|
||||||
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
|
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
|
||||||
@@ -108,11 +115,11 @@ else a `401`.
|
|||||||
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
|
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
|
||||||
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
|
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
|
||||||
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
|
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
|
||||||
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, starred, position |
|
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, kept, position |
|
||||||
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
|
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
|
||||||
| `POST /api/fetch` | |
|
| `POST /api/fetch` | |
|
||||||
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
|
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
|
||||||
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with everyone counted (id, title, art, count, whether it is yours; never a URL, never a private feed); subscribe by id |
|
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with an OPML's feeds in place of the OPML and everyone counted (id, title, art, count, whether it is yours; never a URL, never a private feed); subscribe by id |
|
||||||
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
|
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
|
||||||
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |
|
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |
|
||||||
| `GET /api/events` | SSE, the same broadcast the socket carries |
|
| `GET /api/events` | SSE, the same broadcast the socket carries |
|
||||||
|
|||||||
@@ -60,7 +60,7 @@ ipx reap # actually delete
|
|||||||
```
|
```
|
||||||
|
|
||||||
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
|
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
|
||||||
with no file are pruned from the database. **Starred by anyone keeps a file**, and one only counts
|
with no file are pruned from the database. **An item anyone kept keeps its file**, and one only counts
|
||||||
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
|
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
|
||||||
stops the next scan fetching it again.
|
stops the next scan fetching it again.
|
||||||
|
|
||||||
|
|||||||
@@ -32,10 +32,10 @@ media_types = ["audio", "video"]
|
|||||||
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
|
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
|
||||||
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
|
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
|
||||||
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
|
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
|
||||||
pad. Starred items are never deleted, and a file only counts as read once every subscriber has
|
pad. Kept items are never deleted, and a file only counts as read once every subscriber has
|
||||||
read it. `0` disables it entirely.
|
read it. `0` disables it entirely.
|
||||||
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
|
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
|
||||||
Starred ones stay. `0` disables it.
|
Kept ones stay. `0` disables it.
|
||||||
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
|
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
|
||||||
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
|
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
|
||||||
fetched 216 files and 22 GB in one scan.
|
fetched 216 files and 22 GB in one scan.
|
||||||
@@ -43,8 +43,6 @@ media_types = ["audio", "video"]
|
|||||||
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
|
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
|
||||||
without this the disk fills with artwork. Empty takes everything.
|
without this the disk fills with artwork. Empty takes everything.
|
||||||
|
|
||||||
`interval_mins` from older configs is still read, and `schedule` supersedes it.
|
|
||||||
|
|
||||||
## `[torrent]`
|
## `[torrent]`
|
||||||
|
|
||||||
```toml
|
```toml
|
||||||
@@ -69,6 +67,7 @@ token = "" # generated and saved on first run
|
|||||||
trusted_header = "" # e.g. "Cf-Access-Authenticated-User-Email"
|
trusted_header = "" # e.g. "Cf-Access-Authenticated-User-Email"
|
||||||
trusted_proxies = ["127.0.0.1", "::1"]
|
trusted_proxies = ["127.0.0.1", "::1"]
|
||||||
auto_create_users = true
|
auto_create_users = true
|
||||||
|
sign_out_url = "" # e.g. "/cdn-cgi/access/logout"
|
||||||
session_days = 30
|
session_days = 30
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -79,6 +78,9 @@ session_days = 30
|
|||||||
* **`trusted_proxies`** — addresses allowed to assert that header, and the entire security boundary
|
* **`trusted_proxies`** — addresses allowed to assert that header, and the entire security boundary
|
||||||
for it. Name the proxy, never a subnet.
|
for it. Name the proxy, never a subnet.
|
||||||
* **`auto_create_users`** — create an account the first time the proxy vouches for a new name.
|
* **`auto_create_users`** — create an account the first time the proxy vouches for a new name.
|
||||||
|
* **`sign_out_url`** — where Sign out sends someone the proxy signed in: the proxy's own sign-out,
|
||||||
|
`/cdn-cgi/access/logout` behind Cloudflare Access. Empty sends them to the sign-in page, where
|
||||||
|
the proxy signs them straight back in.
|
||||||
* **`session_days`** — sign a session out after this long without a request.
|
* **`session_days`** — sign a session out after this long without a request.
|
||||||
|
|
||||||
It is plain HTTP. On a LAN bind everything crosses the network in the clear — and a feed URL can
|
It is plain HTTP. On a LAN bind everything crosses the network in the clear — and a feed URL can
|
||||||
|
|||||||
229
docs/history.md
229
docs/history.md
@@ -6,6 +6,235 @@ reasoning lives. New write-ups go at the top.
|
|||||||
|
|
||||||
See [README.md](../README.md) for what the thing is.
|
See [README.md](../README.md) for what the thing is.
|
||||||
|
|
||||||
|
## 2026-09-12 — Healthy while busy
|
||||||
|
|
||||||
|
After a deploy the container sat at "starting" for a minute, and Docker's health log showed two
|
||||||
|
`ipx status` probes exceeding their 5-second timeout. The daemon's own log explained it. The first
|
||||||
|
scan after the start fetched 23 feeds, from 14:10:41 to 14:11:35, and both probes' `status`
|
||||||
|
commands waited in the job queue behind it; they were answered together at 14:11:35, straight after
|
||||||
|
`scan_done`. The worker runs one job at a time and `status` was one of its jobs, so any scan or
|
||||||
|
download longer than about a minute and a half, three 30-second probes, would have had Docker call
|
||||||
|
a working daemon unhealthy.
|
||||||
|
|
||||||
|
The socket now answers `status` itself, from two short queries, and only real work goes through the
|
||||||
|
queue. The trade is that healthy now means the daemon answers on its socket and can read its
|
||||||
|
database; a worker stuck on one job would still pass. Asking a daemon that downloads hour-long
|
||||||
|
podcasts to be idle within five seconds was never a fair test of whether it was alive. A test holds
|
||||||
|
the queue full and checks `status` still comes back.
|
||||||
|
|
||||||
|
The first version broadcast the answer, as the queued one had been. Timing `status` during a forced
|
||||||
|
scan in production showed the catch: `status` is a terminal event, so the `ipx fetch` watching that
|
||||||
|
scan stopped reading at the first probe and printed the status line as its last, while the scan
|
||||||
|
carried on. When `status` waited behind the scan it could never arrive first, so this had never
|
||||||
|
shown. The answer now goes only to the client that asked, and the test checks that another client
|
||||||
|
hears nothing.
|
||||||
|
|
||||||
|
## 2026-09-12 — Signing in through Authentik, for real
|
||||||
|
|
||||||
|
Ray could not get Authentik's sign-in to reach ipx, following `docs/sso.md`, which had been written
|
||||||
|
without ever being tried. Looking at the Cloudflare account through its API showed that side was
|
||||||
|
already complete. Authentik is Zero Trust's OpenID Connect identity provider; the Access application
|
||||||
|
`ipodderx` allows only it and a list of five addresses; the tunnel `rays-unraid` routes
|
||||||
|
`ipodderx.sdf1.net` to `192.168.1.130:8099`; DNS is a proxied CNAME to the tunnel. Access's log
|
||||||
|
showed `rays@sdf1.net` signing in through it. Nothing on Cloudflare was changed, so no other site
|
||||||
|
was touched.
|
||||||
|
|
||||||
|
The gaps were all at ipx's end: `trusted_header` was empty, `trusted_proxies` held only loopback,
|
||||||
|
and the account was called `rays` while the header carries `rays@sdf1.net`.
|
||||||
|
|
||||||
|
Finding the address to trust took the most time. The page said `127.0.0.1`, but `cloudflared` runs in
|
||||||
|
its own container and reaches ipx through the host's published port. ipx logs no peer addresses, so
|
||||||
|
the address was read from `/proc/net/tcp` inside the ipx container: `192.168.16.1`, the gateway of
|
||||||
|
`content_default`, where Docker's masquerade puts traffic crossing from another bridge. A request
|
||||||
|
from Tower's own shell arrived as `192.168.1.130` instead, and a throwaway `busybox` on the default
|
||||||
|
bridge as `192.168.16.1`: the first was refused with the header, the second believed. LAN machines
|
||||||
|
keep their own addresses, since Docker forwards published ports with iptables (the userland proxy
|
||||||
|
only handles loopback).
|
||||||
|
|
||||||
|
Every change, in order, with how to undo it:
|
||||||
|
|
||||||
|
1. **Code**, commit `586d2c0`: `ipx user rename`, deployed. Revert the commit and redeploy to
|
||||||
|
remove it; nothing depends on it once used.
|
||||||
|
2. **Account**: `docker exec iPodderX ipx user rename rays rays@sdf1.net`. Same id, so its feeds,
|
||||||
|
read state, password and admin rights stayed. Undo: `docker exec iPodderX ipx user rename
|
||||||
|
rays@sdf1.net rays`. Signing in at `/login` now takes the new name.
|
||||||
|
3. **Config**, `/mnt/fast/appdata/ipodderx/config.toml`, `[web]`: `trusted_header` from `""` to
|
||||||
|
`"Cf-Access-Authenticated-User-Email"`, and `"192.168.16.1"` added to `trusted_proxies`. The
|
||||||
|
file as it was is `config.toml.2026-09-12-sso.bak` beside it. Undo: copy the backup back and
|
||||||
|
`docker compose -f /mnt/fast/arcane/projects/content/compose.yaml restart ipodderx`.
|
||||||
|
4. **Cloudflare, Docker networks and other containers**: unchanged. The `busybox` test container
|
||||||
|
was removed when it exited, and its image afterwards.
|
||||||
|
5. **Authentik**, later the same day, because ipodderx had no tile in its library while Outline
|
||||||
|
did: a bookmark application `ipodderx` (pk `5854a98e-816a-4c4f-9f27-63e69dc29d1d`), made
|
||||||
|
through the API with a token of Ray's. No provider and no policy bindings, like Outline's, the
|
||||||
|
iPodderX icon, and a link to `https://ipodderx.sdf1.net`. It changes nothing about who can sign
|
||||||
|
in. Undo: delete it under Applications → Applications, or
|
||||||
|
`DELETE /api/v3/core/applications/ipodderx/`.
|
||||||
|
6. **Signing out**, later again. Sign out landed on ipx's password page while Access still vouched
|
||||||
|
for Ray, so it signed nothing out, and the page looked like the wrong login. Cloudflare's
|
||||||
|
`/cdn-cgi/access/logout` ends the Access session for every Access application at once (there is
|
||||||
|
no per-application sign-out, and it takes no redirect), and Authentik's end-session only ends
|
||||||
|
one application's session unless single logout is set up there. Ray chose Access's sign-out. New
|
||||||
|
`[web] sign_out_url`, set to `/cdn-cgi/access/logout` in production (the file as it was is
|
||||||
|
`config.toml.2026-09-12-signout.bak`), and `/login` now sends anyone the proxy vouches for on to
|
||||||
|
`/`. Undo: take the key out and restart; the code does nothing without it.
|
||||||
|
|
||||||
|
What the address trusts is any container on Tower that connects through the host's port, not only
|
||||||
|
`cloudflared`. Verifying Cloudflare's signed `Cf-Access-Jwt-Assertion` would remove that, and is
|
||||||
|
the upgrade if it matters.
|
||||||
|
|
||||||
|
## 2026-09-12 — Trimming the state database
|
||||||
|
|
||||||
|
An audit of the database layer, with a read-only copy of production to check it against. The
|
||||||
|
data was already clean: no tables or indexes left from older versions, 47 free pages after the
|
||||||
|
column drops earlier the same day, and one stray `entry_state` row. The code had five things:
|
||||||
|
|
||||||
|
- `migrate()` still added eight columns to any table missing them. All eight shipped in 0.2.0 and
|
||||||
|
upgrades now start from 0.3.0 at the oldest, so the list and its loop went; the `retired` drop
|
||||||
|
list stays, since a database coming from 0.4.0 still has the old read columns.
|
||||||
|
- `created` on `users`, `subscriptions` and `sessions` was written by every insert and read by
|
||||||
|
nothing. They joined `retired`. The old-database test now builds all three tables, foreign keys
|
||||||
|
included, since `DROP COLUMN` on a table that references another was the part worth proving.
|
||||||
|
- `Db::subscribed_feed_ids` had no callers, though its doc said the scanner walked it.
|
||||||
|
`Db::subscriber_count` had one caller asking whether it was above zero, which
|
||||||
|
`subscriber_counts().contains_key` answers. `Managed.orphaned` was selected and never read.
|
||||||
|
- `users.created` came back the same afternoon, with `last_login` beside it. Nothing read it, but
|
||||||
|
when an account was made and when it last signed in is what you want to know when tidying
|
||||||
|
accounts, and it cannot be recovered later. Both existing accounts got their creation times back
|
||||||
|
from the backup taken before the drop, and a last sign-in from their newest session in it.
|
||||||
|
`last_login` is kept to the hour, because the proxy vouches for every request and that would
|
||||||
|
otherwise be a write each time.
|
||||||
|
|
||||||
|
## 2026-09-12 — Cutting what had outlived its reason
|
||||||
|
|
||||||
|
A whole-repo audit for over-engineering listed twelve things to cut, and all of them went.
|
||||||
|
|
||||||
|
- **Upgrades from before accounts.** `migrate_opml_children` moved OPML feeds that old versions
|
||||||
|
wrote into `config.toml` out to the database, and ran at every daemon start to do nothing after
|
||||||
|
the first. Production ran it in 0.3.0; anything older has to pass through 0.4.0.
|
||||||
|
- **Half of the adoption, and not the other half.** The audit called `adopt_existing_library` a
|
||||||
|
one-time migration and it was cut whole. It did two jobs: copy the old read state into
|
||||||
|
`entry_state`, which was dead, and subscribe the first admin to the whole catalogue while nobody
|
||||||
|
subscribed to anything, which is how a fresh install's first account gets `config.toml`'s feeds.
|
||||||
|
The browser suite caught it at once, signing in to an empty sidebar; `cargo test` had no idea.
|
||||||
|
The second job is back as `adopt_catalogue`, with a unit test of its own.
|
||||||
|
- **The dead `entries` columns.** `read`, `flagged` and `position` moved to `entry_state` with
|
||||||
|
accounts. The adoption's copy was their last reader, but `record_entry` still wrote them, and
|
||||||
|
still reset `read` when a title changed, which nothing looked at. Two bugs came from queries
|
||||||
|
reading them. `migrate()` now drops them from an existing database (SQLite has had `DROP COLUMN`
|
||||||
|
since 3.35), and a test builds an old table to prove it. On production each drop rewrote the
|
||||||
|
66 MB `entries` table, about four seconds apiece, so the first start took thirteen. An
|
||||||
|
`ipx status` run in that window failed with `migrating schema`: every `ipx` command migrates when
|
||||||
|
it opens the database, and it collided with the daemon doing the same. A failed `ALTER TABLE`
|
||||||
|
changes nothing, and the database had been copied to `backup/` first anyway.
|
||||||
|
- **`interval_mins`**, which `schedule` replaced. An old config that still has the key loads; the
|
||||||
|
key is ignored, and the config test carries it to keep that true.
|
||||||
|
- **Three dependencies.** `infer` was only asked whether a file is a torrent, and the check after
|
||||||
|
it already looked for `d8:announce`, which is what `infer` looks for. `dirs` was three lookups of
|
||||||
|
`XDG_CONFIG_HOME`, `XDG_DATA_HOME` and `HOME`. `tokio-stream` wrapped the broadcast receiver for
|
||||||
|
the event stream; `futures_util::stream::unfold` does the same, lagging clients included.
|
||||||
|
- **Two token generators.** The web token came from a copy of the session-token code, with a
|
||||||
|
clock fallback on top. It uses `auth::new_session_token` now, and is 64 characters.
|
||||||
|
- **The icon inlined four times**, 23 KB of base64 each, into both pages. It is `/icon.png` now,
|
||||||
|
outside the sign-in wall with `/login`, since the sign-in page shows it.
|
||||||
|
- Also: the `contrib/` systemd units from before the container, `Db::entries` and
|
||||||
|
`Db::count_entries` that only the tests called, three `logbuf` visitors that repeated the trait's
|
||||||
|
defaults, and unused state, a helper and dead CSS in the page.
|
||||||
|
|
||||||
|
## 2026-09-11 — A design pass on the web UI
|
||||||
|
|
||||||
|
A review against screenshots of every view in all three themes found that Dark and Light read as a
|
||||||
|
generic dark dashboard: one pale blue did every job, most labels were bold, and nothing led. The
|
||||||
|
list it produced, in `TODO.md`, was worked through in one go. What was worth knowing:
|
||||||
|
|
||||||
|
- **Amber means new.** Badges, unread dots and download bars take the icon's EQ amber; blue is left
|
||||||
|
for the primary action and links. Light's amber was `#b06f10`, which gives white text 4.1:1,
|
||||||
|
short of AA for 11 px bold. It is `#9a5f0a` now, 5.2:1.
|
||||||
|
- **EQ bars mark what is playing.** Three `<i>` bars stand at 60, 100 and 40 % and animate only
|
||||||
|
while `body.playing` is set. The first version left the animation on but paused, expecting each
|
||||||
|
bar to hold a different frame. The frames it held were within a pixel of each other, and on
|
||||||
|
screen the bars read as three dots. Under reduced motion one rule drops every animation and
|
||||||
|
transition, which leaves the bars standing.
|
||||||
|
- **The focus ring was clipped.** `.tgroup` and `#topbar` both set `overflow:hidden`, so a ring
|
||||||
|
drawn outside a toolbar button was cut off. Rings inside clipping parents are inset instead.
|
||||||
|
- **The feed list could not be used from the keyboard at all.** Rows were `<div>`s and the triangle
|
||||||
|
a `<span>`, so Tab went from the feed filter to Sign out. Rows now take focus, the triangle is a
|
||||||
|
`<button aria-expanded>`, and `renderFeeds` puts focus back on the same row after redrawing,
|
||||||
|
since every live update replaces every row. The list's own key handler stops Space and the
|
||||||
|
arrows from reaching the player's shortcuts on the document.
|
||||||
|
- **The triangle hangs in the margin.** Every row used to reserve an 18 px slot for it, pushing a
|
||||||
|
hundred feeds 28 px right of the places above for the sake of two folders. It is now absolutely
|
||||||
|
placed in the row's left padding, the full height of the row, so a near miss no longer opens
|
||||||
|
the folder's page.
|
||||||
|
- **A selected tile vanished** because the initials tile and the selected row were both `--raise`.
|
||||||
|
Tiles now mix their tint into `--bg`, which no row uses.
|
||||||
|
- **The Files pane hides itself** with `#split:has(>#files[hidden])`, which collapses its column.
|
||||||
|
The phone layout already hides the pane, and a zero-width extra track there is harmless.
|
||||||
|
- **Flagged became Kept** in the tab, and in the server's refusal to delete a file someone else
|
||||||
|
kept. The filter value and the column stay `flagged`; renaming those buys nothing.
|
||||||
|
|
||||||
|
## 2026-09-11 — A Patreon creator is a list of shows
|
||||||
|
|
||||||
|
Ray asked whether ipx could sync with Patreon. Not in full. The documented API (v2, the
|
||||||
|
`identity.memberships` scope) lists the creators you back and whether each has a feed (`has_rss`),
|
||||||
|
but no resource carries the `auth` token that makes a feed URL work. That token only comes from the
|
||||||
|
creator's page. It is also one per membership, not one per account: techpod's differs from Glass
|
||||||
|
Cannon's, so no single token finds everything you back.
|
||||||
|
|
||||||
|
What does work is one creator at a time, which is what Ray wanted for Glass Cannon and its 33 shows:
|
||||||
|
|
||||||
|
- `patreon.com/rss?auth=<token>`, with no creator named, returns that token's creator. Its self link,
|
||||||
|
about 660 bytes in, gives the campaign by number (`/rss/369921`). Patreon ignores `Range` here, so
|
||||||
|
ipx reads the stream until the number appears and hangs up, instead of taking all 2.8 MB.
|
||||||
|
- A show's `show=` number is a Patreon collection. Asked anonymously, the collection listing
|
||||||
|
(`/api/collection?filter[campaign_id]=`) and a post's `collections` both hide patron-only ones: you
|
||||||
|
get "FAQ". `/api/campaigns/<id>?include=shows` lists every show, anonymously, in one response.
|
||||||
|
- Every spelling works: `rss/glasscannon?auth=…&show=N`, `rss/369921?…` and `rss?auth=…&show=N` all
|
||||||
|
serve the same 131 items. Enclosure URLs are the same in the creator feed and the show feed, and
|
||||||
|
stable between fetches.
|
||||||
|
|
||||||
|
That last point shaped the design. `enclosures.url` is unique, so whichever feed is scanned first owns
|
||||||
|
the file. The first cut only asked a creator for its shows while it had no entries of its own, so that a
|
||||||
|
creator already read as a plain feed, holding every show's episodes, would never be split into shows
|
||||||
|
that came up empty. Within the hour that was the wrong call: Glass Cannon had gone into production on
|
||||||
|
the build before this one, been read as one feed of 2,385 items, and the rule kept it that way. Finding
|
||||||
|
anything in that heap was the problem Ray wanted solved.
|
||||||
|
|
||||||
|
So a creator with more than one show is always a group, run through the same sync as an OPML
|
||||||
|
(`sync_group`, split out of `sync_opml`). When it becomes one, its items are cleared and each show
|
||||||
|
takes over the enclosures the creator holds as the show lists them (`Db::adopt`), downloaded files and
|
||||||
|
everyone's read state included. One show leaves it a plain feed, which is what techpod already was. If
|
||||||
|
the shows cannot be listed, a creator already split fails the scan rather than being read as one heap;
|
||||||
|
one that never was is read as one feed until they can be. An answer without a `shows` list is an error,
|
||||||
|
not "no shows".
|
||||||
|
|
||||||
|
**Filter verdicts follow the settings.** Ray also reported that turning on Allow explicit and
|
||||||
|
rescanning brought nothing back. An item was judged once, when first seen, and `skipped` was final. The
|
||||||
|
2026-09-10 entry below saw it coming ("worth a `ipx retry <feed>` command if this bites"). It bit:
|
||||||
|
2,166 Glass Cannon items and all 88 of Shadowdark's sat at `skipped: explicit` with the setting on.
|
||||||
|
Every scan now runs the filters again over what they skipped (not over `torrents disabled`, which is not
|
||||||
|
a filter's call) and requeues what they now let through. Only that direction: a queued item is never
|
||||||
|
pulled back, because Download latest and a manual download both work by queueing.
|
||||||
|
|
||||||
|
**Two gaps beside it.** Add feed had no explicit box, so every new feed's first scan skipped all its
|
||||||
|
explicit items; it has one now, stored on your subscription like the feed dialog's. And a feed inside a
|
||||||
|
group ignored your settings on the group, though the group's dialog said they were inherited: settings
|
||||||
|
live on each person's subscription, and nothing read the group's. `Db::subscribers` now fills what you
|
||||||
|
have not set on the feed from your subscription to the group, and the feed list shows the same.
|
||||||
|
|
||||||
|
**A name that was already used.** Replaying the split on a copy of the production database left one
|
||||||
|
show with a Supercast episode in it. "Glass Cannon Live! Ascension | Pathfinder 2E" slugs to
|
||||||
|
`glass-cannon-live-ascension-pathfinder-2`, the id of a Supercast feed of the same show that had been
|
||||||
|
removed. Removing a feed keeps its rows on purpose, so that re-adding it does not fetch the back
|
||||||
|
catalogue again, but choosing a new id only checked config.toml and derived feeds. The Patreon show took
|
||||||
|
the old id and everything still filed under it. An id is now also taken when the database has a feed by
|
||||||
|
that id at a different URL; the same URL may still have it back, which is the re-add case.
|
||||||
|
|
||||||
|
Shows already added by hand are matched by token and show number, not by exact URL (`same_feed`), so a
|
||||||
|
bare token does not add Get in the Trunk and Shadowdark a second time under another spelling.
|
||||||
|
|
||||||
|
The show listing is Patreon's own undocumented web API. If it changes, only finding new shows stops.
|
||||||
|
|
||||||
## 2026-09-11 — One meaning per icon, sortable columns, and one player
|
## 2026-09-11 — One meaning per icon, sortable columns, and one player
|
||||||
|
|
||||||
Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act
|
Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act
|
||||||
|
|||||||
307
docs/sso.md
307
docs/sso.md
@@ -1,9 +1,8 @@
|
|||||||
# Signing in through Cloudflare Zero Trust or Authentik
|
# Signing in through Cloudflare Access and Authentik
|
||||||
|
|
||||||
ipx can take the signed-in identity from whatever sits in front of it, instead of asking for a
|
ipx can take the signed-in identity from whatever sits in front of it, instead of asking for a
|
||||||
password itself. Both products below do the same thing in the end: they authenticate the person and
|
password itself. The proxy authenticates the person and passes the result to ipx in a **header**;
|
||||||
pass the result to the origin in a **header**. ipx reads that header, finds (or creates) the
|
ipx reads it, finds (or creates) the matching account, and gets on with it.
|
||||||
matching account, and gets on with it.
|
|
||||||
|
|
||||||
Read [How this is secured](#how-this-is-secured) before exposing anything. The short version: a
|
Read [How this is secured](#how-this-is-secured) before exposing anything. The short version: a
|
||||||
header is worth exactly as much as the hop that set it, so ipx only believes one from an address you
|
header is worth exactly as much as the hop that set it, so ipx only believes one from an address you
|
||||||
@@ -11,195 +10,173 @@ list.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## The ipx side (both setups)
|
## How ipodderx.sdf1.net does it
|
||||||
|
|
||||||
|
Checked end to end on 2026-09-12. An earlier version of this page had never been tried against a
|
||||||
|
real setup and pointed at the wrong address.
|
||||||
|
|
||||||
|
```
|
||||||
|
browser ─► Cloudflare Access, app "ipodderx" ─── sign in ───► Authentik (OpenID Connect)
|
||||||
|
─► tunnel "rays-unraid" (the cloudflared container on Tower)
|
||||||
|
─► http://192.168.1.130:8099 ─► ipx
|
||||||
|
```
|
||||||
|
|
||||||
|
Authentik is not in the request path. It is the identity provider Cloudflare Access asks. Access
|
||||||
|
then adds `Cf-Access-Authenticated-User-Email`, the email address Authentik gave it, to every
|
||||||
|
request it forwards through the tunnel, and ipx signs that person in.
|
||||||
|
|
||||||
|
| Piece | Where | Setting |
|
||||||
|
|---|---|---|
|
||||||
|
| Identity provider | Zero Trust → Settings → Authentication | `Authentik`, OpenID Connect; scopes `openid email profile` |
|
||||||
|
| Access application | Zero Trust → Access → Applications → `ipodderx` | Domain `ipodderx.sdf1.net`; identity providers: Authentik only, with instant auth; session 730h; policy *Require Login* allows a list of email addresses |
|
||||||
|
| Tunnel route | Zero Trust → Networks → Tunnels → `rays-unraid` → Public hostnames | `ipodderx.sdf1.net` → HTTP `192.168.1.130:8099` |
|
||||||
|
| DNS | `sdf1.net` | `ipodderx` CNAME to the tunnel, proxied |
|
||||||
|
| ipx | `/mnt/fast/appdata/ipodderx/config.toml`, `[web]` | below |
|
||||||
|
|
||||||
```toml
|
```toml
|
||||||
[web]
|
[web]
|
||||||
enabled = true
|
enabled = true
|
||||||
bind = "0.0.0.0:8099"
|
bind = "0.0.0.0:8099"
|
||||||
token = "…" # keep it: it is the admin, used by the healthcheck
|
trusted_header = "Cf-Access-Authenticated-User-Email"
|
||||||
|
trusted_proxies = ["127.0.0.1", "::1", "192.168.16.1"]
|
||||||
# The header your proxy sets. Empty (the default) disables this whole path.
|
|
||||||
trusted_header = "Cf-Access-Authenticated-User-Email" # Authentik: "X-authentik-username"
|
|
||||||
|
|
||||||
# Addresses allowed to assert that header -- the proxy, and nothing else.
|
|
||||||
trusted_proxies = ["127.0.0.1", "::1"]
|
|
||||||
|
|
||||||
# Create an account the first time the proxy vouches for a name ipx has not seen.
|
|
||||||
auto_create_users = true
|
auto_create_users = true
|
||||||
|
sign_out_url = "/cdn-cgi/access/logout"
|
||||||
session_days = 30
|
session_days = 30
|
||||||
```
|
```
|
||||||
|
|
||||||
Restart the daemon after editing. Accounts made this way have **no password**: they can only ever
|
Restart ipx after editing it: `docker compose -f /mnt/fast/arcane/projects/content/compose.yaml
|
||||||
arrive through the proxy. `ipx user list` marks them `proxy only`.
|
restart ipodderx`.
|
||||||
|
|
||||||
The first account created is an admin. Every later one is an ordinary user, and an ordinary user
|
### What was missing
|
||||||
cannot change global settings, a feed's URL or folder, or how often feeds are scanned: the API
|
|
||||||
refuses those with a `403`, not just the UI. Everything else about a feed (which items they want,
|
|
||||||
whether to fetch them, how many at a time) is theirs alone; see [users.md](users.md).
|
|
||||||
|
|
||||||
Somebody arriving through the proxy for the first time starts with **no feeds**, because
|
Cloudflare and Authentik were already right. Three things on the ipx side were not:
|
||||||
subscriptions are per person. Adding a feed someone else already reads costs no second fetch and no
|
|
||||||
second copy on disk.
|
|
||||||
|
|
||||||
Promote someone with:
|
1. **`trusted_header` was empty**, which switches the whole proxy path off. ipx ignored the header
|
||||||
|
and asked for a password.
|
||||||
|
2. **`trusted_proxies` listed only `127.0.0.1`.** The tunnel's requests do not come from there;
|
||||||
|
see the next section.
|
||||||
|
3. **The account had the wrong name.** It was made by hand as `rays`, but the header carries
|
||||||
|
`rays@sdf1.net`. With `auto_create_users` on, the first visit would have made a second, empty
|
||||||
|
account. `ipx user rename rays rays@sdf1.net` fixed that without losing anything.
|
||||||
|
|
||||||
|
### The address to trust, and why it is 192.168.16.1
|
||||||
|
|
||||||
|
`cloudflared` runs in its own container and reaches ipx through the host's published port. Docker
|
||||||
|
(iptables firewall backend) masquerades traffic between its bridge networks, so the tunnel's
|
||||||
|
requests arrive from the **gateway of ipx's own network**, `content_default`:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
ipx user list
|
docker network inspect content_default -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'
|
||||||
echo -n 'a good password' | ipx user passwd <name> # optional: also lets them sign in directly
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Local sign-in at `/login` keeps working alongside all of this, which is how you get in from the LAN
|
That was measured, not assumed. ipx does not log where a request came from, so the addresses were
|
||||||
when the tunnel is down. So does the shared `[web] token`, which signs in as the admin: that is
|
read from the kernel's connection table inside the container while the site was open. (`/proc/net/tcp`
|
||||||
what the Docker healthcheck uses, and the way back in if you lock yourself out. A brand new database
|
lists them in hex.)
|
||||||
starts with **admin / ipodderx** — change it.
|
|
||||||
|
If the `content` project's network is ever recreated, its gateway can change. Check it again, and
|
||||||
|
update `trusted_proxies` to match.
|
||||||
|
|
||||||
|
### Names
|
||||||
|
|
||||||
|
The username is the email address, lower-cased: `rays@sdf1.net`. To sign in at `/login` with a
|
||||||
|
password from the LAN, use that name too.
|
||||||
|
|
||||||
|
To let someone else in, add their address to the Access policy; they need an Authentik account with
|
||||||
|
that email. With `auto_create_users = true` they get an ipx account on their first visit, as an
|
||||||
|
ordinary user with no feeds. An account made before the proxy can be given the name the proxy will
|
||||||
|
send:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker exec iPodderX ipx user rename <old name> <email address>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Signing out
|
||||||
|
|
||||||
|
**Sign out** sends someone the proxy signed in to `sign_out_url`, here Cloudflare's
|
||||||
|
`/cdn-cgi/access/logout`. That ends your Access session for **every** Access application,
|
||||||
|
`code.sdf1.net` included: Cloudflare has no way to end just one, and its sign-out page does not send
|
||||||
|
you anywhere afterwards. The next visit goes back through Authentik, which lets you straight in if
|
||||||
|
you are still signed in there. Signing out of Authentik itself is Authentik's own sign-out.
|
||||||
|
|
||||||
|
ipx never shows its password page to someone the proxy vouches for: `/login` sends them on to their
|
||||||
|
feeds.
|
||||||
|
|
||||||
|
### The tile in Authentik's library
|
||||||
|
|
||||||
|
Authentik's library lists Authentik's own applications, and ipodderx signs in through the one
|
||||||
|
called `Cloudflare Access`, so ipodderx needs a bookmark of its own to show up there. It is
|
||||||
|
Applications → Applications → `ipodderx`: no provider, launch URL `https://ipodderx.sdf1.net`, and
|
||||||
|
the iPodderX icon. Like Outline's, it has no policy bindings, so everyone in Authentik sees the
|
||||||
|
tile. Who actually gets in is still up to the Access policy.
|
||||||
|
|
||||||
|
### Check it
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# From Tower itself: not a trusted address, so the header is ignored.
|
||||||
|
curl -s -H 'Accept: application/json' -H 'Cf-Access-Authenticated-User-Email: rays@sdf1.net' \
|
||||||
|
http://192.168.1.130:8099/api/me # -> sign in
|
||||||
|
|
||||||
|
# From a container on a Docker bridge, as cloudflared is: believed.
|
||||||
|
docker run --rm --network bridge mirror.gcr.io/library/busybox wget -qO- \
|
||||||
|
--header 'Accept: application/json' --header 'Cf-Access-Authenticated-User-Email: rays@sdf1.net' \
|
||||||
|
http://192.168.1.130:8099/api/me # -> {"admin":true,"name":"rays@sdf1.net"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Then open `https://ipodderx.sdf1.net` in a private window. Authentik should ask who you are, and
|
||||||
|
ipx should show `rays@sdf1.net` in the sidebar footer without asking for a password.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Cloudflare Zero Trust
|
## The ipx settings
|
||||||
|
|
||||||
This is what runs `ipodderx.sdf1.net`: a `cloudflared` tunnel to the origin, with an Access
|
| Key | What it does |
|
||||||
application in front of it. Cloudflare authenticates the visitor and adds
|
|
||||||
`Cf-Access-Authenticated-User-Email` to every request it forwards.
|
|
||||||
|
|
||||||
### 1. The tunnel
|
|
||||||
|
|
||||||
In **Zero Trust → Networks → Tunnels**, either use the existing tunnel or create one, then add a
|
|
||||||
public hostname:
|
|
||||||
|
|
||||||
| Field | Value |
|
|
||||||
|---|---|
|
|---|---|
|
||||||
| Subdomain / domain | `ipodderx` / `sdf1.net` |
|
| `trusted_header` | The header the proxy sets. Empty, the default, turns the proxy path off. |
|
||||||
| Type | HTTP |
|
| `trusted_proxies` | The addresses allowed to set it. Nothing else is believed. |
|
||||||
| URL | `localhost:8099` (or the LAN address of the box) |
|
| `auto_create_users` | Make an account the first time the proxy vouches for a name ipx has not seen. |
|
||||||
|
| `sign_out_url` | Where Sign out sends someone the proxy signed in: the proxy's own sign-out. Empty sends them to the sign-in page, where the proxy signs them straight back in. |
|
||||||
|
| `session_days` | How long a password sign-in lasts without use. |
|
||||||
|
|
||||||
Use `localhost` when `cloudflared` runs on the same machine as ipx — that keeps the origin request
|
The first account ever created is an admin. Every later one is an ordinary user, who cannot change
|
||||||
coming from `127.0.0.1`, which is already in `trusted_proxies`. If `cloudflared` runs elsewhere (its
|
global settings, a feed's URL or folder, or how often feeds are scanned: the API refuses those with
|
||||||
own container, another host), put **its** address in `trusted_proxies` instead, and make sure
|
a `403`, not just the UI. Everything else about a feed is theirs alone; see [users.md](users.md).
|
||||||
nothing else can reach port 8099.
|
|
||||||
|
|
||||||
### 2. The Access application
|
Local sign-in at `/login` keeps working alongside the proxy, which is how you get in from the LAN
|
||||||
|
when the tunnel is down. So does the shared `[web] token`, which signs in as the admin and is the
|
||||||
**Zero Trust → Access → Applications → Add an application → Self-hosted**:
|
way back in if you lock yourself out. A brand new database starts with **admin / ipodderx**;
|
||||||
|
change it.
|
||||||
- Application domain: `ipodderx.sdf1.net`
|
|
||||||
- Session duration: whatever suits; ipx keeps its own 30-day session on top.
|
|
||||||
- Add a policy — *Allow*, with a rule such as `Emails` → your address, or `Emails ending in` →
|
|
||||||
your domain. Anyone this policy admits gets an ipx account when `auto_create_users` is on, so keep
|
|
||||||
the policy as narrow as the people you actually want reading your feeds.
|
|
||||||
|
|
||||||
### 3. Point ipx at the header
|
|
||||||
|
|
||||||
```toml
|
|
||||||
trusted_header = "Cf-Access-Authenticated-User-Email"
|
|
||||||
trusted_proxies = ["127.0.0.1", "::1"]
|
|
||||||
```
|
|
||||||
|
|
||||||
The username becomes the email address, lower-cased (`ray@example.com`). That is what shows in the
|
|
||||||
sidebar and what `ipx user list` prints.
|
|
||||||
|
|
||||||
### 4. Check it
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# From the box itself: no header, no session -> the sign-in page.
|
|
||||||
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' -H 'Accept: text/html' http://127.0.0.1:8099/
|
|
||||||
|
|
||||||
# Pretending to be the tunnel (only works because 127.0.0.1 is trusted):
|
|
||||||
curl -s -H 'Cf-Access-Authenticated-User-Email: you@example.com' http://127.0.0.1:8099/api/me
|
|
||||||
```
|
|
||||||
|
|
||||||
Then load `https://ipodderx.sdf1.net` in a browser: Cloudflare should ask who you are, and ipx
|
|
||||||
should show your address in the sidebar footer without ever asking for a password.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Authentik
|
## Authentik in the request path instead
|
||||||
|
|
||||||
Authentik does this with a **Proxy Provider** plus an **outpost**, which sits in the request path and
|
Not what ipodderx.sdf1.net uses, and **not verified**. Authentik can also sit in front of ipx
|
||||||
adds `X-authentik-username` (also `X-authentik-email`, `X-authentik-name`, `X-authentik-groups`).
|
itself, with a **Proxy Provider** and an **outpost** that adds `X-authentik-username`:
|
||||||
|
|
||||||
### 1. Provider
|
- Applications → Providers → Create → Proxy Provider; mode **Proxy** (the outpost talks to ipx) or
|
||||||
|
**Forward auth** (an existing reverse proxy asks the outpost).
|
||||||
**Applications → Providers → Create → Proxy Provider**:
|
- Applications → Create, bound to that provider, with a policy; add the provider to an outpost.
|
||||||
|
- In ipx: `trusted_header = "X-authentik-username"`, and the outpost's or reverse proxy's address
|
||||||
- Name: `ipx`
|
in `trusted_proxies`. Measure that address as above rather than guessing it.
|
||||||
- Authorization flow: your usual (`default-provider-authorization-implicit-consent`)
|
|
||||||
- Mode: **Forward auth (single application)** if an existing reverse proxy fronts ipx, or
|
|
||||||
**Proxy** to let the outpost talk to ipx directly.
|
|
||||||
- External host: `https://ipodderx.example.net`
|
|
||||||
- Internal host (Proxy mode): `http://<ip of the ipx box>:8099`
|
|
||||||
|
|
||||||
### 2. Application and outpost
|
|
||||||
|
|
||||||
**Applications → Create**, bind it to that provider, and give it a policy so only the people you
|
|
||||||
mean are let through. Then add the provider to an outpost (**Applications → Outposts**, the embedded
|
|
||||||
one is fine).
|
|
||||||
|
|
||||||
### 3. Forward auth, if you use nginx/SWAG in front
|
|
||||||
|
|
||||||
In the server block for ipx:
|
|
||||||
|
|
||||||
```nginx
|
|
||||||
location /outpost.goauthentik.io {
|
|
||||||
proxy_pass http://authentik-server:9000/outpost.goauthentik.io;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
|
||||||
add_header Set-Cookie $auth_cookie;
|
|
||||||
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
|
||||||
}
|
|
||||||
|
|
||||||
location / {
|
|
||||||
auth_request /outpost.goauthentik.io/auth/nginx;
|
|
||||||
error_page 401 = @goauthentik_proxy_signin;
|
|
||||||
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
|
||||||
add_header Set-Cookie $auth_cookie;
|
|
||||||
|
|
||||||
# This is the line that matters to ipx.
|
|
||||||
auth_request_set $authentik_username $upstream_http_x_authentik_username;
|
|
||||||
proxy_set_header X-authentik-username $authentik_username;
|
|
||||||
|
|
||||||
proxy_pass http://ipx:8099;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Point ipx at the header
|
|
||||||
|
|
||||||
```toml
|
|
||||||
trusted_header = "X-authentik-username"
|
|
||||||
trusted_proxies = ["172.18.0.5"] # the outpost or nginx container, NOT a whole subnet
|
|
||||||
```
|
|
||||||
|
|
||||||
Usernames arrive as Authentik knows them (`ray`), lower-cased.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## How this is secured
|
## How this is secured
|
||||||
|
|
||||||
**The header is only believed from `trusted_proxies`.** Every other source is ignored, and the
|
**The header is only believed from `trusted_proxies`.** Every other source is ignored, and the
|
||||||
request falls through to a session cookie or the shared token. This is the whole security boundary,
|
request falls through to a session cookie or the shared token. That is the whole security boundary.
|
||||||
so:
|
|
||||||
|
|
||||||
- List the **proxy's own address**, not a range. `["127.0.0.1"]` when the tunnel runs beside ipx;
|
With the tunnel reaching ipx through the host's port, `192.168.16.1` means **any container on Tower
|
||||||
the container's IP when it does not.
|
that connects to `192.168.1.130:8099`**, not only `cloudflared`. Machines on the LAN, and Tower
|
||||||
- Never list a LAN subnet. Anyone on your network could then send
|
itself, arrive under their own addresses and cannot set the header; the checks above show both
|
||||||
`Cf-Access-Authenticated-User-Email: admin@…` and be your admin.
|
sides. Never list a LAN address or range: anyone there could then send
|
||||||
- Make sure the origin port is not reachable *around* the proxy by anyone you would not admit
|
`Cf-Access-Authenticated-User-Email: rays@sdf1.net` and be you.
|
||||||
through it. If it is, bind ipx to `127.0.0.1` and let only the proxy reach it.
|
|
||||||
|
|
||||||
Verify the refusal, don't assume it — set `trusted_proxies = ["10.9.9.9"]` briefly and confirm a
|
**What ipx does not do:** it does not verify Cloudflare's signed `Cf-Access-Jwt-Assertion`. It
|
||||||
header from your machine gets a `401`:
|
trusts the hop. Verifying the signature would make the containers on Tower irrelevant to the
|
||||||
|
boundary, and is the upgrade if that ever matters.
|
||||||
|
|
||||||
```sh
|
**Turning it off:** clear `trusted_header` and restart. Proxy-made accounts stay, but nobody can sign
|
||||||
curl -s -o /dev/null -w '%{http_code}\n' \
|
|
||||||
-H 'Cf-Access-Authenticated-User-Email: someone@example.com' http://127.0.0.1:8099/api/me
|
|
||||||
```
|
|
||||||
|
|
||||||
**What ipx does not do:** it does not verify Cloudflare's `Cf-Access-Jwt-Assertion` signature or
|
|
||||||
Authentik's session. It trusts the hop. That is a deliberate trade — it keeps the configuration to
|
|
||||||
three lines — and it is sound exactly as long as the point above holds.
|
|
||||||
|
|
||||||
**Turning it off:** clear `trusted_header`. Existing proxy-only accounts stay, but nobody can sign
|
|
||||||
in with them until they are given a password (`ipx user passwd <name>`).
|
in with them until they are given a password (`ipx user passwd <name>`).
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -207,22 +184,20 @@ in with them until they are given a password (`ipx user passwd <name>`).
|
|||||||
## Everyday administration
|
## Everyday administration
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
ipx user list # who exists, and how each one signs in
|
ipx user list # who exists, how each signs in, and when
|
||||||
echo -n 'secret123' | ipx user add sam # local account, password on stdin
|
echo -n 'secret123' | ipx user add sam # local account, password on stdin
|
||||||
ipx user add sam --no-password # proxy-only account, created ahead of time
|
ipx user add sam@example.com --no-password # proxy-only account, made ahead of time
|
||||||
|
ipx user rename sam sam@example.com # give an account the name the proxy sends
|
||||||
echo -n 'newsecret' | ipx user passwd sam # change a password
|
echo -n 'newsecret' | ipx user passwd sam # change a password
|
||||||
ipx user rm sam # remove the account
|
ipx user rm sam # remove the account
|
||||||
```
|
```
|
||||||
|
|
||||||
Set `auto_create_users = false` once everyone who should have an account has one. After that the
|
In the container, put `docker exec iPodderX` in front, and `docker exec -i iPodderX` for the ones
|
||||||
proxy vouching for an unknown name is logged and refused, rather than quietly making an account.
|
that read a password.
|
||||||
Pre-create people instead with `ipx user add <name> --no-password`, using exactly the name the
|
|
||||||
header will carry (Cloudflare sends the email address, lower-cased).
|
|
||||||
|
|
||||||
Scanning intervals, the disk quota, retention, the download folder and a feed's URL are
|
Set `auto_create_users = false` once everyone who should have an account has one. After that the
|
||||||
**admin-only**: the Settings button is hidden for everyone else, and the API refuses the change even
|
proxy vouching for an unknown name is logged and refused. Make people ahead of time instead, with
|
||||||
if the request is made by hand. Everyone controls their own keywords, auto-download, explicit
|
the exact name the header will carry.
|
||||||
setting and per-scan cap, along with their own read state and which feeds they see.
|
|
||||||
|
|
||||||
See also [users.md](users.md) for what several people share, [configuration.md](configuration.md)
|
See also [users.md](users.md) for what several people share, [configuration.md](configuration.md)
|
||||||
for every `[web]` key, and [cli.md](cli.md) for the `ipx user` commands.
|
for every `[web]` key, and [cli.md](cli.md) for the `ipx user` commands.
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ fetch, one parse and one file.
|
|||||||
|
|
||||||
| Yours alone | The same for everyone |
|
| Yours alone | The same for everyone |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Read, starred, playback position | The feed's URL |
|
| Read, kept, playback position | The feed's URL |
|
||||||
| Which feeds you see at all | Its download folder |
|
| Which feeds you see at all | Its download folder |
|
||||||
| Keywords, auto-download, explicit, per-scan cap | When it is scanned |
|
| Keywords, auto-download, explicit, per-scan cap | When it is scanned |
|
||||||
| | The file on disk |
|
| | The file on disk |
|
||||||
@@ -34,10 +34,10 @@ is shared.
|
|||||||
|
|
||||||
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
|
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
|
||||||
for everyone** and names them in the confirmation, and the server has the last word: if anyone else
|
for everyone** and names them in the confirmation, and the server has the last word: if anyone else
|
||||||
has starred the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
|
has kept the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
|
||||||
reason, and only `?force=true` goes through.
|
reason, and only `?force=true` goes through.
|
||||||
|
|
||||||
Retention follows the same rule: starred by anyone keeps a file, and it counts as read only once
|
Retention follows the same rule: an item anyone kept keeps its file, and it counts as read only once
|
||||||
every subscriber has read it.
|
every subscriber has read it.
|
||||||
|
|
||||||
## Signing in
|
## Signing in
|
||||||
@@ -73,12 +73,13 @@ re-subscribing does not pull the back catalogue again.
|
|||||||
**Popular** and **Directory** sit at the top of the feed list, above your own feeds. Popular, also
|
**Popular** and **Directory** sit at the top of the feed list, above your own feeds. Popular, also
|
||||||
shown in the Add feed dialog, lists the ten feeds with the most subscribers on this server, you
|
shown in the Add feed dialog, lists the ten feeds with the most subscribers on this server, you
|
||||||
included. Directory lists every one of them A to Z. Your own feeds are marked Subscribed.
|
included. Directory lists every one of them A to Z. Your own feeds are marked Subscribed.
|
||||||
It shows a title, artwork and a count, never a URL or who reads it. Feeds from an
|
It shows a title, artwork and a count, never a URL or who reads it. An OPML subscription is listed
|
||||||
OPML subscription are left out, since they come with the OPML. So is anything that looks private: a
|
as the feeds inside it, one by one, and never the OPML itself, so you can take just the shows you
|
||||||
login configured for the feed, credentials in its URL, or a key such as `auth=` or `token=` in the
|
want. Anything that looks private is left out: a login configured for the feed, credentials in its URL,
|
||||||
query, or a feed from a paid-feed service such as Patreon or Supercast, which put the key in the
|
or a key such as `auth=` or `token=` in the query, or a feed from a paid-feed service such as
|
||||||
path. Those are someone's paid subscriptions, and listing them would let anyone here read what they
|
Patreon or Supercast, which put the key in the path, and any feed inside an OPML that looks private
|
||||||
pay for.
|
itself. Those are someone's paid subscriptions, and listing them would let anyone here read what
|
||||||
|
they pay for.
|
||||||
|
|
||||||
An admin can do the same from **Settings → Manage users…**: add someone (with a password, or none
|
An admin can do the same from **Settings → Manage users…**: add someone (with a password, or none
|
||||||
for someone the proxy signs in), tick or untick Admin, or remove an account. Removing one takes its
|
for someone the proxy signs in), tick or untick Admin, or remove an account. Removing one takes its
|
||||||
|
|||||||
@@ -26,9 +26,6 @@ pub struct General {
|
|||||||
/// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d".
|
/// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d".
|
||||||
/// A feed's own `schedule` overrides this.
|
/// A feed's own `schedule` overrides this.
|
||||||
pub schedule: String,
|
pub schedule: String,
|
||||||
/// Superseded by `schedule`. Still read so existing configs keep working.
|
|
||||||
#[serde(skip_serializing_if = "Option::is_none")]
|
|
||||||
pub interval_mins: Option<u64>,
|
|
||||||
pub organize: Organize,
|
pub organize: Organize,
|
||||||
/// 0 = unlimited.
|
/// 0 = unlimited.
|
||||||
pub max_total_gb: f64,
|
pub max_total_gb: f64,
|
||||||
@@ -83,6 +80,10 @@ pub struct Web {
|
|||||||
pub trusted_proxies: Vec<String>,
|
pub trusted_proxies: Vec<String>,
|
||||||
/// Create an account the first time the proxy vouches for a name it has not seen.
|
/// Create an account the first time the proxy vouches for a name it has not seen.
|
||||||
pub auto_create_users: bool,
|
pub auto_create_users: bool,
|
||||||
|
/// Where Sign out sends someone the proxy signed in. Signing out of ipx alone cannot stick
|
||||||
|
/// while the proxy still vouches for them, so this is the proxy's own sign-out:
|
||||||
|
/// `/cdn-cgi/access/logout` behind Cloudflare Access. Empty sends them to /login.
|
||||||
|
pub sign_out_url: String,
|
||||||
/// Sign a session out after this long without a request.
|
/// Sign a session out after this long without a request.
|
||||||
pub session_days: i64,
|
pub session_days: i64,
|
||||||
}
|
}
|
||||||
@@ -96,6 +97,7 @@ impl Default for Web {
|
|||||||
trusted_header: String::new(),
|
trusted_header: String::new(),
|
||||||
trusted_proxies: vec!["127.0.0.1".into(), "::1".into()],
|
trusted_proxies: vec!["127.0.0.1".into(), "::1".into()],
|
||||||
auto_create_users: true,
|
auto_create_users: true,
|
||||||
|
sign_out_url: String::new(),
|
||||||
session_days: 30,
|
session_days: 30,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -153,7 +155,6 @@ impl Default for General {
|
|||||||
download_dir: home().join("Podcasts"),
|
download_dir: home().join("Podcasts"),
|
||||||
socket: default_socket(),
|
socket: default_socket(),
|
||||||
schedule: "every 60m".into(),
|
schedule: "every 60m".into(),
|
||||||
interval_mins: None,
|
|
||||||
organize: Organize::Feed,
|
organize: Organize::Feed,
|
||||||
max_total_gb: 0.0,
|
max_total_gb: 0.0,
|
||||||
max_age_days: 0,
|
max_age_days: 0,
|
||||||
@@ -176,8 +177,8 @@ impl Default for Torrent {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl General {
|
impl General {
|
||||||
/// Minutes between checks. Falls back to the legacy `interval_mins`, then to an hour.
|
/// Minutes between checks, or an hour when `schedule` is empty or unreadable. A malformed
|
||||||
/// A malformed value warns rather than stopping the daemon.
|
/// value warns rather than stopping the daemon.
|
||||||
pub fn interval(&self) -> u64 {
|
pub fn interval(&self) -> u64 {
|
||||||
if let Some(n) = parse_interval(&self.schedule) {
|
if let Some(n) = parse_interval(&self.schedule) {
|
||||||
return n;
|
return n;
|
||||||
@@ -185,7 +186,7 @@ impl General {
|
|||||||
if !self.schedule.trim().is_empty() {
|
if !self.schedule.trim().is_empty() {
|
||||||
tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default");
|
tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default");
|
||||||
}
|
}
|
||||||
self.interval_mins.filter(|n| *n > 0).unwrap_or(60)
|
60
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -278,9 +279,7 @@ pub fn config_path() -> PathBuf {
|
|||||||
if let Ok(p) = std::env::var("IPX_CONFIG") {
|
if let Ok(p) = std::env::var("IPX_CONFIG") {
|
||||||
return PathBuf::from(p);
|
return PathBuf::from(p);
|
||||||
}
|
}
|
||||||
dirs::config_dir()
|
xdg("XDG_CONFIG_HOME", ".config").join("ipx/config.toml")
|
||||||
.unwrap_or_else(|| home().join(".config"))
|
|
||||||
.join("ipx/config.toml")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`.
|
/// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`.
|
||||||
@@ -288,9 +287,7 @@ pub fn data_dir() -> PathBuf {
|
|||||||
if let Ok(p) = std::env::var("IPX_DATA_DIR") {
|
if let Ok(p) = std::env::var("IPX_DATA_DIR") {
|
||||||
return PathBuf::from(p);
|
return PathBuf::from(p);
|
||||||
}
|
}
|
||||||
dirs::data_dir()
|
xdg("XDG_DATA_HOME", ".local/share").join("ipx")
|
||||||
.unwrap_or_else(|| home().join(".local/share"))
|
|
||||||
.join("ipx")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn default_socket() -> PathBuf {
|
fn default_socket() -> PathBuf {
|
||||||
@@ -346,8 +343,16 @@ pub fn unique_slug(text: &str, taken: &BTreeMap<String, Feed>) -> String {
|
|||||||
(2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap()
|
(2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `$var`, or `~/fallback` when it is unset or empty, as the XDG base directory spec says.
|
||||||
|
fn xdg(var: &str, fallback: &str) -> PathBuf {
|
||||||
|
std::env::var_os(var)
|
||||||
|
.filter(|v| !v.is_empty())
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|| home().join(fallback))
|
||||||
|
}
|
||||||
|
|
||||||
fn home() -> PathBuf {
|
fn home() -> PathBuf {
|
||||||
dirs::home_dir().unwrap_or_else(|| PathBuf::from("."))
|
std::env::var_os("HOME").map(PathBuf::from).unwrap_or_else(|| PathBuf::from("."))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn expand_tilde(p: &Path) -> PathBuf {
|
fn expand_tilde(p: &Path) -> PathBuf {
|
||||||
@@ -367,6 +372,8 @@ mod tests {
|
|||||||
r#"
|
r#"
|
||||||
[general]
|
[general]
|
||||||
download_dir = "/tmp/pods"
|
download_dir = "/tmp/pods"
|
||||||
|
# A key older versions read. An old config that still has it has to load.
|
||||||
|
interval_mins = 45
|
||||||
|
|
||||||
[feeds.example]
|
[feeds.example]
|
||||||
url = "https://example.com/feed.xml"
|
url = "https://example.com/feed.xml"
|
||||||
@@ -403,22 +410,17 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn interval_falls_back_through_legacy_then_default() {
|
fn interval_falls_back_to_an_hour() {
|
||||||
let mut g = General::default();
|
let mut g = General::default();
|
||||||
assert_eq!(g.interval(), 60, "the default schedule");
|
assert_eq!(g.interval(), 60, "the default schedule");
|
||||||
|
|
||||||
g.schedule = "every 15m".into();
|
g.schedule = "every 15m".into();
|
||||||
assert_eq!(g.interval(), 15);
|
assert_eq!(g.interval(), 15);
|
||||||
|
|
||||||
// A config written before `schedule` existed still works.
|
// Empty or garbage must not stop the daemon.
|
||||||
g.schedule = String::new();
|
g.schedule = String::new();
|
||||||
g.interval_mins = Some(45);
|
assert_eq!(g.interval(), 60);
|
||||||
assert_eq!(g.interval(), 45);
|
|
||||||
|
|
||||||
// Garbage must not stop the daemon.
|
|
||||||
g.schedule = "whenever".into();
|
g.schedule = "whenever".into();
|
||||||
assert_eq!(g.interval(), 45);
|
|
||||||
g.interval_mins = None;
|
|
||||||
assert_eq!(g.interval(), 60);
|
assert_eq!(g.interval(), 60);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
511
src/db.rs
511
src/db.rs
@@ -40,14 +40,10 @@ CREATE TABLE IF NOT EXISTS entries (
|
|||||||
published INTEGER,
|
published INTEGER,
|
||||||
description TEXT,
|
description TEXT,
|
||||||
first_seen INTEGER NOT NULL,
|
first_seen INTEGER NOT NULL,
|
||||||
read INTEGER NOT NULL DEFAULT 0,
|
|
||||||
flagged INTEGER NOT NULL DEFAULT 0,
|
|
||||||
image TEXT,
|
image TEXT,
|
||||||
duration INTEGER,
|
duration INTEGER,
|
||||||
episode INTEGER,
|
episode INTEGER,
|
||||||
season INTEGER,
|
season INTEGER,
|
||||||
-- Seconds into the audio, so playback resumes where it was left.
|
|
||||||
position INTEGER NOT NULL DEFAULT 0,
|
|
||||||
PRIMARY KEY (feed_id, guid)
|
PRIMARY KEY (feed_id, guid)
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -77,7 +73,9 @@ CREATE TABLE IF NOT EXISTS users (
|
|||||||
name TEXT NOT NULL UNIQUE COLLATE NOCASE,
|
name TEXT NOT NULL UNIQUE COLLATE NOCASE,
|
||||||
pass_hash TEXT,
|
pass_hash TEXT,
|
||||||
is_admin INTEGER NOT NULL DEFAULT 0,
|
is_admin INTEGER NOT NULL DEFAULT 0,
|
||||||
created INTEGER NOT NULL
|
-- For whoever maintains the server. NULL where it is not known.
|
||||||
|
created INTEGER,
|
||||||
|
last_login INTEGER
|
||||||
);
|
);
|
||||||
|
|
||||||
-- What one person wants from a feed. The feed, its items and its files are shared; this
|
-- What one person wants from a feed. The feed, its items and its files are shared; this
|
||||||
@@ -89,11 +87,10 @@ CREATE TABLE IF NOT EXISTS subscriptions (
|
|||||||
auto_download INTEGER,
|
auto_download INTEGER,
|
||||||
allow_explicit INTEGER,
|
allow_explicit INTEGER,
|
||||||
max_new_per_check INTEGER,
|
max_new_per_check INTEGER,
|
||||||
created INTEGER NOT NULL,
|
|
||||||
PRIMARY KEY (user_id, feed_id)
|
PRIMARY KEY (user_id, feed_id)
|
||||||
);
|
);
|
||||||
|
|
||||||
-- Read, starred and how far in. One row per person per item, created on first touch;
|
-- Read, kept and how far in. One row per person per item, created on first touch;
|
||||||
-- an item nobody has touched has no row at all, which is what unread means.
|
-- an item nobody has touched has no row at all, which is what unread means.
|
||||||
CREATE TABLE IF NOT EXISTS entry_state (
|
CREATE TABLE IF NOT EXISTS entry_state (
|
||||||
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||||
@@ -108,7 +105,6 @@ CREATE TABLE IF NOT EXISTS entry_state (
|
|||||||
CREATE TABLE IF NOT EXISTS sessions (
|
CREATE TABLE IF NOT EXISTS sessions (
|
||||||
token TEXT PRIMARY KEY,
|
token TEXT PRIMARY KEY,
|
||||||
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||||
created INTEGER NOT NULL,
|
|
||||||
seen INTEGER NOT NULL
|
seen INTEGER NOT NULL
|
||||||
);
|
);
|
||||||
";
|
";
|
||||||
@@ -131,6 +127,23 @@ pub struct User {
|
|||||||
pub name: String,
|
pub name: String,
|
||||||
pub pass_hash: Option<String>,
|
pub pass_hash: Option<String>,
|
||||||
pub is_admin: bool,
|
pub is_admin: bool,
|
||||||
|
/// When the account was made and when it last signed in, for whoever maintains the server.
|
||||||
|
pub created: Option<i64>,
|
||||||
|
pub last_login: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The columns `user_row` reads, in its order.
|
||||||
|
const USER_COLS: &str = "id, name, pass_hash, is_admin, created, last_login";
|
||||||
|
|
||||||
|
fn user_row(r: &rusqlite::Row<'_>) -> rusqlite::Result<User> {
|
||||||
|
Ok(User {
|
||||||
|
id: r.get(0)?,
|
||||||
|
name: r.get(1)?,
|
||||||
|
pass_hash: r.get(2)?,
|
||||||
|
is_admin: r.get::<_, i64>(3)? != 0,
|
||||||
|
created: r.get(4)?,
|
||||||
|
last_login: r.get(5)?,
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A feed derived from an OPML subscription rather than written into the config.
|
/// A feed derived from an OPML subscription rather than written into the config.
|
||||||
@@ -140,33 +153,48 @@ pub struct Managed {
|
|||||||
pub url: String,
|
pub url: String,
|
||||||
pub title: Option<String>,
|
pub title: Option<String>,
|
||||||
pub group_id: String,
|
pub group_id: String,
|
||||||
pub orphaned: bool,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Adds columns that later versions introduced. CREATE TABLE IF NOT EXISTS does nothing to
|
/// Adds the columns later versions introduced and drops the ones they retired. CREATE TABLE IF
|
||||||
/// a table that already exists, so an installed database needs them added explicitly.
|
/// NOT EXISTS leaves a table that already exists alone, so an installed database needs both done
|
||||||
|
/// explicitly. Columns from before 0.3.0, the oldest version an upgrade may start from, need no
|
||||||
|
/// entry.
|
||||||
fn migrate(conn: &Connection) -> Result<()> {
|
fn migrate(conn: &Connection) -> Result<()> {
|
||||||
let wanted: &[(&str, &str, &str)] = &[
|
let wanted: &[(&str, &str, &str)] = &[
|
||||||
("feeds", "image", "TEXT"),
|
// For whoever maintains the server. An audit dropped `created` as unread on 2026-09-12,
|
||||||
("feeds", "orphaned", "INTEGER NOT NULL DEFAULT 0"),
|
// and it came back the same day with `last_login` beside it.
|
||||||
("feeds", "group_id", "TEXT"),
|
("users", "created", "INTEGER"),
|
||||||
("feeds", "managed", "INTEGER NOT NULL DEFAULT 0"),
|
("users", "last_login", "INTEGER"),
|
||||||
("entries", "image", "TEXT"),
|
|
||||||
("entries", "duration", "INTEGER"),
|
|
||||||
("entries", "episode", "INTEGER"),
|
|
||||||
("entries", "season", "INTEGER"),
|
|
||||||
("entries", "position", "INTEGER NOT NULL DEFAULT 0"),
|
|
||||||
];
|
];
|
||||||
|
let retired: &[(&str, &str)] = &[
|
||||||
|
// Read state from before accounts, long since moved to entry_state. Two bugs came from
|
||||||
|
// queries still reading these after they stopped meaning anything.
|
||||||
|
("entries", "read"),
|
||||||
|
("entries", "flagged"),
|
||||||
|
("entries", "position"),
|
||||||
|
// Written by every insert and read by nothing.
|
||||||
|
("subscriptions", "created"),
|
||||||
|
("sessions", "created"),
|
||||||
|
];
|
||||||
|
let has = |table: &str, column: &str| -> Result<bool> {
|
||||||
|
let names: Vec<String> = conn
|
||||||
|
.prepare(&format!("PRAGMA table_info({table})"))?
|
||||||
|
.query_map([], |r| r.get(1))?
|
||||||
|
.collect::<rusqlite::Result<_>>()?;
|
||||||
|
Ok(names.iter().any(|c| c == column))
|
||||||
|
};
|
||||||
for (table, column, ty) in wanted {
|
for (table, column, ty) in wanted {
|
||||||
let mut stmt = conn.prepare(&format!("PRAGMA table_info({table})"))?;
|
if !has(table, column)? {
|
||||||
let existing: Vec<String> = stmt
|
|
||||||
.query_map([], |r| r.get::<_, String>(1))?
|
|
||||||
.collect::<rusqlite::Result<Vec<_>>>()?;
|
|
||||||
if !existing.iter().any(|c| c == column) {
|
|
||||||
tracing::info!(table, column, "adding column");
|
tracing::info!(table, column, "adding column");
|
||||||
conn.execute_batch(&format!("ALTER TABLE {table} ADD COLUMN {column} {ty}"))?;
|
conn.execute_batch(&format!("ALTER TABLE {table} ADD COLUMN {column} {ty}"))?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for (table, column) in retired {
|
||||||
|
if has(table, column)? {
|
||||||
|
tracing::info!(table, column, "dropping column");
|
||||||
|
conn.execute_batch(&format!("ALTER TABLE {table} DROP COLUMN {column}"))?;
|
||||||
|
}
|
||||||
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -345,22 +373,19 @@ impl Db {
|
|||||||
|
|
||||||
/// Returns true when this entry had not been seen before.
|
/// Returns true when this entry had not been seen before.
|
||||||
///
|
///
|
||||||
/// A changed description or title flips `read` back to 0, which is what the original's
|
|
||||||
/// textDiff dance was ultimately for -- minus the diff markup, which the UI can do.
|
|
||||||
pub fn record_entry(&self, feed_id: &str, e: &crate::feed::Entry) -> Result<bool> {
|
pub fn record_entry(&self, feed_id: &str, e: &crate::feed::Entry) -> Result<bool> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let inserted = conn.execute(
|
let inserted = conn.execute(
|
||||||
"INSERT OR IGNORE INTO entries
|
"INSERT OR IGNORE INTO entries
|
||||||
(feed_id, guid, title, link, published, description, first_seen, read, flagged,
|
(feed_id, guid, title, link, published, description, first_seen,
|
||||||
image, duration, episode, season)
|
image, duration, episode, season)
|
||||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0, 0, ?8, ?9, ?10, ?11)",
|
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)",
|
||||||
rusqlite::params![
|
rusqlite::params![
|
||||||
feed_id, e.guid, e.title, e.link, e.published, e.description, now(),
|
feed_id, e.guid, e.title, e.link, e.published, e.description, now(),
|
||||||
e.image, e.duration, e.episode, e.season
|
e.image, e.duration, e.episode, e.season
|
||||||
],
|
],
|
||||||
)?;
|
)?;
|
||||||
if inserted == 0 {
|
if inserted == 0 {
|
||||||
// The SET expressions see the pre-update row, so this compares old vs new.
|
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"UPDATE entries SET
|
"UPDATE entries SET
|
||||||
title = coalesce(?3, title),
|
title = coalesce(?3, title),
|
||||||
@@ -368,8 +393,7 @@ impl Db {
|
|||||||
image = coalesce(?5, image),
|
image = coalesce(?5, image),
|
||||||
duration = coalesce(?6, duration),
|
duration = coalesce(?6, duration),
|
||||||
episode = coalesce(?7, episode),
|
episode = coalesce(?7, episode),
|
||||||
season = coalesce(?8, season),
|
season = coalesce(?8, season)
|
||||||
read = CASE WHEN description IS NOT ?4 OR title IS NOT ?3 THEN 0 ELSE read END
|
|
||||||
WHERE feed_id = ?1 AND guid = ?2",
|
WHERE feed_id = ?1 AND guid = ?2",
|
||||||
rusqlite::params![
|
rusqlite::params![
|
||||||
feed_id, e.guid, e.title, e.description,
|
feed_id, e.guid, e.title, e.description,
|
||||||
@@ -687,22 +711,9 @@ pub struct EncRow {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl Db {
|
impl Db {
|
||||||
/// One page of a feed's entries, newest first, each with its enclosures attached.
|
/// One page of entries, each with its enclosures attached: one feed's, or every feed the
|
||||||
/// `search` matches title and description, case-insensitively.
|
/// person subscribes to when `feed_id` is None (All Subscriptions). `search` matches title
|
||||||
pub fn entries(
|
/// and description, case-insensitively.
|
||||||
&self,
|
|
||||||
user_id: i64,
|
|
||||||
feed_id: &str,
|
|
||||||
filter: Filter,
|
|
||||||
search: Option<&str>,
|
|
||||||
offset: i64,
|
|
||||||
limit: i64,
|
|
||||||
) -> Result<Vec<EntryRow>> {
|
|
||||||
self.entries_in(user_id, Some(feed_id), filter, search, offset, limit, &order_sql("published", "desc"))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `entries` for one feed, or across every feed the person subscribes to when `feed_id`
|
|
||||||
/// is None: the All Subscriptions view.
|
|
||||||
pub fn entries_in(
|
pub fn entries_in(
|
||||||
&self,
|
&self,
|
||||||
user_id: i64,
|
user_id: i64,
|
||||||
@@ -793,18 +804,7 @@ impl Db {
|
|||||||
Ok(rows)
|
Ok(rows)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// How many entries match, so the UI knows whether there is another page.
|
/// How many entries `entries_in` would page through, so the UI knows whether there is more.
|
||||||
pub fn count_entries(
|
|
||||||
&self,
|
|
||||||
user_id: i64,
|
|
||||||
feed_id: &str,
|
|
||||||
filter: Filter,
|
|
||||||
search: Option<&str>,
|
|
||||||
) -> Result<i64> {
|
|
||||||
self.count_in(user_id, Some(feed_id), filter, search)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `count_entries` for one feed, or across every feed the person subscribes to.
|
|
||||||
pub fn count_in(
|
pub fn count_in(
|
||||||
&self,
|
&self,
|
||||||
user_id: i64,
|
user_id: i64,
|
||||||
@@ -838,37 +838,28 @@ impl Db {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Marks every entry in a feed read, for the "mark all read" button.
|
/// The first admin starts subscribed to the whole catalogue: whoever wrote config.toml meant
|
||||||
/// Moves a single-user library onto an account: everything read, starred or part-played
|
/// to read those feeds, and without this a fresh install signs in to an empty sidebar. Runs
|
||||||
/// becomes that person's, and they subscribe to every feed already in the catalogue.
|
/// only while nobody subscribes to anything, so an unsubscribe is never undone.
|
||||||
/// Runs once -- the moment there is a first account and no subscriptions yet.
|
pub fn adopt_catalogue(&self, user_id: i64, catalogue: &[String]) -> Result<usize> {
|
||||||
pub fn adopt_existing_library(&self, user_id: i64, catalogue: &[String]) -> Result<usize> {
|
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let already: i64 =
|
let already: i64 =
|
||||||
conn.query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0))?;
|
conn.query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0))?;
|
||||||
if already > 0 {
|
if already > 0 {
|
||||||
return Ok(0);
|
return Ok(0);
|
||||||
}
|
}
|
||||||
let moved = conn.execute(
|
|
||||||
"INSERT INTO entry_state (user_id, feed_id, guid, read, flagged, position)
|
|
||||||
SELECT ?1, feed_id, guid, read, flagged, position FROM entries
|
|
||||||
WHERE read = 1 OR flagged = 1 OR position > 0
|
|
||||||
ON CONFLICT(user_id, feed_id, guid) DO NOTHING",
|
|
||||||
[user_id],
|
|
||||||
)?;
|
|
||||||
for id in catalogue {
|
for id in catalogue {
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created) VALUES (?1, ?2, ?3)",
|
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id) VALUES (?1, ?2)",
|
||||||
params![user_id, id, now()],
|
params![user_id, id],
|
||||||
)?;
|
)?;
|
||||||
}
|
}
|
||||||
// Feeds that exist only in the database (OPML children) count too.
|
// Feeds that exist only in the database (OPML children) count too.
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created)
|
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id) SELECT ?1, id FROM feeds",
|
||||||
SELECT ?1, id, ?2 FROM feeds",
|
params![user_id],
|
||||||
params![user_id, now()],
|
|
||||||
)?;
|
)?;
|
||||||
Ok(moved)
|
Ok(catalogue.len())
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- subscriptions ----
|
// ---- subscriptions ----
|
||||||
@@ -920,14 +911,20 @@ impl Db {
|
|||||||
|
|
||||||
/// Everyone's settings for one feed. The scanner merges these into what it fetches
|
/// Everyone's settings for one feed. The scanner merges these into what it fetches
|
||||||
/// and downloads, since one file serves the lot.
|
/// and downloads, since one file serves the lot.
|
||||||
pub fn subscribers(&self, feed_id: &str) -> Result<Vec<Sub>> {
|
/// In a group, whatever someone has not set on the feed itself comes from their
|
||||||
|
/// subscription to the group, as the group's settings dialog has always said it does.
|
||||||
|
pub fn subscribers(&self, feed_id: &str, group: Option<&str>) -> Result<Vec<Sub>> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let mut stmt = conn.prepare(
|
let mut stmt = conn.prepare(
|
||||||
"SELECT keywords, auto_download, allow_explicit, max_new_per_check
|
"SELECT coalesce(c.keywords, p.keywords), coalesce(c.auto_download, p.auto_download),
|
||||||
FROM subscriptions WHERE feed_id = ?1",
|
coalesce(c.allow_explicit, p.allow_explicit),
|
||||||
|
coalesce(c.max_new_per_check, p.max_new_per_check)
|
||||||
|
FROM subscriptions c
|
||||||
|
LEFT JOIN subscriptions p ON p.user_id = c.user_id AND p.feed_id = ?2
|
||||||
|
WHERE c.feed_id = ?1",
|
||||||
)?;
|
)?;
|
||||||
let out = stmt
|
let out = stmt
|
||||||
.query_map([feed_id], |r| {
|
.query_map(params![feed_id, group], |r| {
|
||||||
Ok(Sub {
|
Ok(Sub {
|
||||||
feed_id: feed_id.to_string(),
|
feed_id: feed_id.to_string(),
|
||||||
keywords: r
|
keywords: r
|
||||||
@@ -977,8 +974,8 @@ impl Db {
|
|||||||
pub fn subscribe(&self, user_id: i64, feed_id: &str) -> Result<()> {
|
pub fn subscribe(&self, user_id: i64, feed_id: &str) -> Result<()> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created) VALUES (?1, ?2, ?3)",
|
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id) VALUES (?1, ?2)",
|
||||||
params![user_id, feed_id, now()],
|
params![user_id, feed_id],
|
||||||
)?;
|
)?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -992,16 +989,6 @@ impl Db {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// How many people want this feed. Nobody means it stops being scanned.
|
|
||||||
pub fn subscriber_count(&self, feed_id: &str) -> Result<i64> {
|
|
||||||
let conn = self.conn.lock().unwrap();
|
|
||||||
Ok(conn.query_row(
|
|
||||||
"SELECT count(*) FROM subscriptions WHERE feed_id = ?1",
|
|
||||||
[feed_id],
|
|
||||||
|r| r.get(0),
|
|
||||||
)?)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Overwrites one person's settings for a feed. A None field means: follow the feed.
|
/// Overwrites one person's settings for a feed. A None field means: follow the feed.
|
||||||
pub fn set_subscription(&self, user_id: i64, sub: &Sub) -> Result<()> {
|
pub fn set_subscription(&self, user_id: i64, sub: &Sub) -> Result<()> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
@@ -1012,8 +999,8 @@ impl Db {
|
|||||||
.transpose()?;
|
.transpose()?;
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT INTO subscriptions
|
"INSERT INTO subscriptions
|
||||||
(user_id, feed_id, keywords, auto_download, allow_explicit, max_new_per_check, created)
|
(user_id, feed_id, keywords, auto_download, allow_explicit, max_new_per_check)
|
||||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
|
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
|
||||||
ON CONFLICT(user_id, feed_id) DO UPDATE SET
|
ON CONFLICT(user_id, feed_id) DO UPDATE SET
|
||||||
keywords = excluded.keywords,
|
keywords = excluded.keywords,
|
||||||
auto_download = excluded.auto_download,
|
auto_download = excluded.auto_download,
|
||||||
@@ -1026,22 +1013,11 @@ impl Db {
|
|||||||
sub.auto_download.map(|v| v as i64),
|
sub.auto_download.map(|v| v as i64),
|
||||||
sub.allow_explicit.map(|v| v as i64),
|
sub.allow_explicit.map(|v| v as i64),
|
||||||
sub.max_new_per_check,
|
sub.max_new_per_check,
|
||||||
now()
|
|
||||||
],
|
],
|
||||||
)?;
|
)?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Feeds with at least one subscriber. What the scanner walks.
|
|
||||||
pub fn subscribed_feed_ids(&self) -> Result<Vec<String>> {
|
|
||||||
let conn = self.conn.lock().unwrap();
|
|
||||||
let mut stmt = conn.prepare("SELECT DISTINCT feed_id FROM subscriptions")?;
|
|
||||||
let out = stmt
|
|
||||||
.query_map([], |r| r.get::<_, String>(0))?
|
|
||||||
.collect::<rusqlite::Result<Vec<_>>>()?;
|
|
||||||
Ok(out)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---- users and sessions ----
|
// ---- users and sessions ----
|
||||||
|
|
||||||
pub fn create_user(&self, name: &str, pass_hash: Option<&str>, admin: bool) -> Result<i64> {
|
pub fn create_user(&self, name: &str, pass_hash: Option<&str>, admin: bool) -> Result<i64> {
|
||||||
@@ -1054,11 +1030,11 @@ impl Db {
|
|||||||
}
|
}
|
||||||
|
|
||||||
pub fn user_by_name(&self, name: &str) -> Result<Option<User>> {
|
pub fn user_by_name(&self, name: &str) -> Result<Option<User>> {
|
||||||
self.one_user("SELECT id, name, pass_hash, is_admin FROM users WHERE name = ?1", name)
|
self.one_user(&format!("SELECT {USER_COLS} FROM users WHERE name = ?1"), name)
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn user_by_id(&self, id: i64) -> Result<Option<User>> {
|
pub fn user_by_id(&self, id: i64) -> Result<Option<User>> {
|
||||||
self.one_user("SELECT id, name, pass_hash, is_admin FROM users WHERE id = ?1", id)
|
self.one_user(&format!("SELECT {USER_COLS} FROM users WHERE id = ?1"), id)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn one_user<P: rusqlite::ToSql>(&self, sql: &str, key: P) -> Result<Option<User>> {
|
fn one_user<P: rusqlite::ToSql>(&self, sql: &str, key: P) -> Result<Option<User>> {
|
||||||
@@ -1066,12 +1042,7 @@ impl Db {
|
|||||||
let mut stmt = conn.prepare(sql)?;
|
let mut stmt = conn.prepare(sql)?;
|
||||||
let mut rows = stmt.query(params![key])?;
|
let mut rows = stmt.query(params![key])?;
|
||||||
Ok(match rows.next()? {
|
Ok(match rows.next()? {
|
||||||
Some(r) => Some(User {
|
Some(r) => Some(user_row(r)?),
|
||||||
id: r.get(0)?,
|
|
||||||
name: r.get(1)?,
|
|
||||||
pass_hash: r.get(2)?,
|
|
||||||
is_admin: r.get::<_, i64>(3)? != 0,
|
|
||||||
}),
|
|
||||||
None => None,
|
None => None,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -1079,16 +1050,9 @@ impl Db {
|
|||||||
pub fn users(&self) -> Result<Vec<User>> {
|
pub fn users(&self) -> Result<Vec<User>> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let mut stmt =
|
let mut stmt =
|
||||||
conn.prepare("SELECT id, name, pass_hash, is_admin FROM users ORDER BY name")?;
|
conn.prepare(&format!("SELECT {USER_COLS} FROM users ORDER BY name"))?;
|
||||||
let out = stmt
|
let out = stmt
|
||||||
.query_map([], |r| {
|
.query_map([], user_row)?
|
||||||
Ok(User {
|
|
||||||
id: r.get(0)?,
|
|
||||||
name: r.get(1)?,
|
|
||||||
pass_hash: r.get(2)?,
|
|
||||||
is_admin: r.get::<_, i64>(3)? != 0,
|
|
||||||
})
|
|
||||||
})?
|
|
||||||
.collect::<rusqlite::Result<Vec<_>>>()?;
|
.collect::<rusqlite::Result<Vec<_>>>()?;
|
||||||
Ok(out)
|
Ok(out)
|
||||||
}
|
}
|
||||||
@@ -1105,6 +1069,26 @@ impl Db {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The proxy signs people in by the name it vouches for, so an account made before the proxy
|
||||||
|
/// was set up has to take that name to be found by it. The name is UNIQUE, so a taken one is
|
||||||
|
/// refused here as well as by the caller.
|
||||||
|
pub fn rename_user(&self, id: i64, name: &str) -> Result<()> {
|
||||||
|
let conn = self.conn.lock().unwrap();
|
||||||
|
conn.execute("UPDATE users SET name = ?2 WHERE id = ?1", params![id, name])?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records a sign-in, to the hour: the proxy vouches for every request, and writing each one
|
||||||
|
/// would buy nothing.
|
||||||
|
pub fn signed_in(&self, id: i64) -> Result<()> {
|
||||||
|
let conn = self.conn.lock().unwrap();
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE users SET last_login = ?2 WHERE id = ?1 AND coalesce(last_login, 0) <= ?2 - 3600",
|
||||||
|
params![id, now()],
|
||||||
|
)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
/// Sessions go with the user: a deleted account must not leave a usable cookie behind.
|
/// Sessions go with the user: a deleted account must not leave a usable cookie behind.
|
||||||
pub fn delete_user(&self, id: i64) -> Result<()> {
|
pub fn delete_user(&self, id: i64) -> Result<()> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
@@ -1116,7 +1100,7 @@ impl Db {
|
|||||||
pub fn create_session(&self, user_id: i64, token: &str) -> Result<()> {
|
pub fn create_session(&self, user_id: i64, token: &str) -> Result<()> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT INTO sessions (token, user_id, created, seen) VALUES (?1, ?2, ?3, ?3)",
|
"INSERT INTO sessions (token, user_id, seen) VALUES (?1, ?2, ?3)",
|
||||||
params![token, user_id, now()],
|
params![token, user_id, now()],
|
||||||
)?;
|
)?;
|
||||||
Ok(())
|
Ok(())
|
||||||
@@ -1128,18 +1112,13 @@ impl Db {
|
|||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let cutoff = now() - max_idle_secs;
|
let cutoff = now() - max_idle_secs;
|
||||||
let mut stmt = conn.prepare(
|
let mut stmt = conn.prepare(
|
||||||
"SELECT u.id, u.name, u.pass_hash, u.is_admin
|
"SELECT u.id, u.name, u.pass_hash, u.is_admin, u.created, u.last_login
|
||||||
FROM sessions s JOIN users u ON u.id = s.user_id
|
FROM sessions s JOIN users u ON u.id = s.user_id
|
||||||
WHERE s.token = ?1 AND s.seen >= ?2",
|
WHERE s.token = ?1 AND s.seen >= ?2",
|
||||||
)?;
|
)?;
|
||||||
let mut rows = stmt.query(params![token, cutoff])?;
|
let mut rows = stmt.query(params![token, cutoff])?;
|
||||||
let found = match rows.next()? {
|
let found = match rows.next()? {
|
||||||
Some(r) => Some(User {
|
Some(r) => Some(user_row(r)?),
|
||||||
id: r.get(0)?,
|
|
||||||
name: r.get(1)?,
|
|
||||||
pass_hash: r.get(2)?,
|
|
||||||
is_admin: r.get::<_, i64>(3)? != 0,
|
|
||||||
}),
|
|
||||||
None => None,
|
None => None,
|
||||||
};
|
};
|
||||||
drop(rows);
|
drop(rows);
|
||||||
@@ -1218,7 +1197,7 @@ impl Db {
|
|||||||
.optional()?)
|
.optional()?)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Read and starred, per person. The row is created on first touch.
|
/// Read and kept, per person. The row is created on first touch.
|
||||||
pub fn set_entry_flag(
|
pub fn set_entry_flag(
|
||||||
&self,
|
&self,
|
||||||
user_id: i64,
|
user_id: i64,
|
||||||
@@ -1293,7 +1272,7 @@ impl Db {
|
|||||||
pub fn managed_feeds(&self) -> Result<Vec<Managed>> {
|
pub fn managed_feeds(&self) -> Result<Vec<Managed>> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
let mut stmt = conn.prepare(
|
let mut stmt = conn.prepare(
|
||||||
"SELECT id, url, title, group_id, orphaned FROM feeds
|
"SELECT id, url, title, group_id FROM feeds
|
||||||
WHERE managed = 1 AND group_id IS NOT NULL ORDER BY coalesce(title, id)",
|
WHERE managed = 1 AND group_id IS NOT NULL ORDER BY coalesce(title, id)",
|
||||||
)?;
|
)?;
|
||||||
Ok(stmt
|
Ok(stmt
|
||||||
@@ -1303,7 +1282,6 @@ impl Db {
|
|||||||
url: r.get(1)?,
|
url: r.get(1)?,
|
||||||
title: r.get(2)?,
|
title: r.get(2)?,
|
||||||
group_id: r.get(3)?,
|
group_id: r.get(3)?,
|
||||||
orphaned: r.get::<_, i64>(4)? != 0,
|
|
||||||
})
|
})
|
||||||
})?
|
})?
|
||||||
.collect::<rusqlite::Result<Vec<_>>>()?)
|
.collect::<rusqlite::Result<Vec<_>>>()?)
|
||||||
@@ -1325,6 +1303,71 @@ impl Db {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Empties a feed of its items, leaving its files alone.
|
||||||
|
pub fn clear_entries(&self, feed_id: &str) -> Result<()> {
|
||||||
|
let conn = self.conn.lock().unwrap();
|
||||||
|
conn.execute("DELETE FROM entries WHERE feed_id = ?1", [feed_id])?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every feed the database holds rows for, as (id, url), removed ones included.
|
||||||
|
pub fn feed_urls(&self) -> Result<Vec<(String, String)>> {
|
||||||
|
let conn = self.conn.lock().unwrap();
|
||||||
|
let mut stmt = conn.prepare("SELECT id, coalesce(url, '') FROM feeds")?;
|
||||||
|
let out = stmt
|
||||||
|
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))?
|
||||||
|
.collect::<rusqlite::Result<_>>()?;
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hands a feed in a group the enclosures its parent holds, as (guid, url), with everyone's
|
||||||
|
/// read state for them. A Patreon creator read as one feed before it was split into shows
|
||||||
|
/// owns every show's files, and `enclosures.url` is unique, so without this each show would
|
||||||
|
/// list its items with nothing to play.
|
||||||
|
pub fn adopt(&self, parent: &str, child: &str, listed: &[(&str, &str)]) -> Result<()> {
|
||||||
|
let mut conn = self.conn.lock().unwrap();
|
||||||
|
let holds: bool = conn.query_row(
|
||||||
|
"SELECT EXISTS (SELECT 1 FROM enclosures WHERE feed_id = ?1)",
|
||||||
|
[parent],
|
||||||
|
|r| r.get(0),
|
||||||
|
)?;
|
||||||
|
if !holds {
|
||||||
|
return Ok(()); // An OPML, or a creator already shared out.
|
||||||
|
}
|
||||||
|
let tx = conn.transaction()?;
|
||||||
|
for &(guid, url) in listed {
|
||||||
|
let moved = tx.execute(
|
||||||
|
"UPDATE enclosures SET feed_id = ?3, guid = ?4 WHERE url = ?1 AND feed_id = ?2",
|
||||||
|
params![url, parent, child, guid],
|
||||||
|
)?;
|
||||||
|
if moved == 1 {
|
||||||
|
// Patreon gives a post the same guid in every feed it appears in.
|
||||||
|
tx.execute(
|
||||||
|
"UPDATE OR IGNORE entry_state SET feed_id = ?2 WHERE feed_id = ?1 AND guid = ?3",
|
||||||
|
params![parent, child, guid],
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tx.commit()?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A feed's enclosures skipped by one of its filters, by URL, with the reason: the verdicts a
|
||||||
|
/// change of settings can overturn. A torrent held back while torrents are off is not a
|
||||||
|
/// filter's call.
|
||||||
|
pub fn skipped_by_filter(&self, feed_id: &str) -> Result<std::collections::HashMap<String, String>> {
|
||||||
|
let conn = self.conn.lock().unwrap();
|
||||||
|
let mut stmt = conn.prepare(
|
||||||
|
"SELECT url, last_error FROM enclosures
|
||||||
|
WHERE feed_id = ?1 AND state = 'skipped' AND last_error IS NOT NULL
|
||||||
|
AND last_error != 'torrents disabled'",
|
||||||
|
)?;
|
||||||
|
let out = stmt
|
||||||
|
.query_map([feed_id], |r| Ok((r.get(0)?, r.get(1)?)))?
|
||||||
|
.collect::<rusqlite::Result<_>>()?;
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn set_orphaned(&self, feed_id: &str, on: bool) -> Result<()> {
|
pub fn set_orphaned(&self, feed_id: &str, on: bool) -> Result<()> {
|
||||||
let conn = self.conn.lock().unwrap();
|
let conn = self.conn.lock().unwrap();
|
||||||
conn.execute(
|
conn.execute(
|
||||||
@@ -1381,8 +1424,8 @@ mod tests {
|
|||||||
fn every_sort_column_runs_and_orders_both_ways() {
|
fn every_sort_column_runs_and_orders_both_ways() {
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().unwrap();
|
||||||
db.exec_for_test(
|
db.exec_for_test(
|
||||||
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0);
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
|
||||||
INSERT INTO subscriptions (user_id, feed_id, created) VALUES (1,'f',0),(1,'g',0);
|
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(1,'g');
|
||||||
INSERT INTO feeds (id, url, title) VALUES ('f','u','Zebra'),('g','v','Aardvark');
|
INSERT INTO feeds (id, url, title) VALUES ('f','u','Zebra'),('g','v','Aardvark');
|
||||||
INSERT INTO entries (feed_id, guid, title, first_seen) VALUES
|
INSERT INTO entries (feed_id, guid, title, first_seen) VALUES
|
||||||
('f','a','banana',100),('g','b','Apple',200),('f','c','cherry',300);
|
('f','a','banana',100),('g','b','Apple',200),('f','c','cherry',300);
|
||||||
@@ -1415,8 +1458,8 @@ mod tests {
|
|||||||
fn deleting_a_shared_file_asks_about_everyone_else() {
|
fn deleting_a_shared_file_asks_about_everyone_else() {
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().unwrap();
|
||||||
db.exec_for_test(
|
db.exec_for_test(
|
||||||
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0),(3,'kit',0,0);
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0),(3,'kit',0);
|
||||||
INSERT INTO subscriptions (user_id, feed_id, created) VALUES (1,'f',0),(2,'f',0),(3,'f',0);
|
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f'),(3,'f');
|
||||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES ('f','a',0);
|
INSERT INTO entries (feed_id, guid, first_seen) VALUES ('f','a',0);
|
||||||
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
|
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
|
||||||
(1,'f','a','u1','/tmp/a','done');",
|
(1,'f','a','u1','/tmp/a','done');",
|
||||||
@@ -1441,7 +1484,7 @@ mod tests {
|
|||||||
fn read_state_belongs_to_one_person() {
|
fn read_state_belongs_to_one_person() {
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().unwrap();
|
||||||
db.exec_for_test(
|
db.exec_for_test(
|
||||||
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0);
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
|
||||||
INSERT INTO entries (feed_id, guid, title, first_seen) VALUES
|
INSERT INTO entries (feed_id, guid, title, first_seen) VALUES
|
||||||
('f','a','One',100),('f','b','Two',200);",
|
('f','a','One',100),('f','b','Two',200);",
|
||||||
)
|
)
|
||||||
@@ -1457,8 +1500,9 @@ mod tests {
|
|||||||
// Starring and position are just as private.
|
// Starring and position are just as private.
|
||||||
db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).unwrap();
|
db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).unwrap();
|
||||||
db.set_position(2, "f", "b", 42).unwrap();
|
db.set_position(2, "f", "b", 42).unwrap();
|
||||||
let ray = db.entries(1, "f", Filter::All, None, 0, 50).unwrap();
|
let order = order_sql("published", "desc");
|
||||||
let sam = db.entries(2, "f", Filter::All, None, 0, 50).unwrap();
|
let page = |user| db.entries_in(user, Some("f"), Filter::All, None, 0, 50, &order).unwrap();
|
||||||
|
let (ray, sam) = (page(1), page(2));
|
||||||
let ray_b = ray.iter().find(|e| e.guid == "b").unwrap();
|
let ray_b = ray.iter().find(|e| e.guid == "b").unwrap();
|
||||||
let sam_b = sam.iter().find(|e| e.guid == "b").unwrap();
|
let sam_b = sam.iter().find(|e| e.guid == "b").unwrap();
|
||||||
assert!(ray_b.flagged && ray_b.position == 0);
|
assert!(ray_b.flagged && ray_b.position == 0);
|
||||||
@@ -1482,6 +1526,111 @@ mod tests {
|
|||||||
assert_eq!(sum.downloaded, 0);
|
assert_eq!(sum.downloaded, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_old_database_loses_its_retired_columns() {
|
||||||
|
let conn = Connection::open_in_memory().unwrap();
|
||||||
|
// As open() has it: a DROP COLUMN on a table that references another is the part worth
|
||||||
|
// proving, and it has to work with the foreign keys switched on.
|
||||||
|
conn.pragma_update(None, "foreign_keys", "ON").unwrap();
|
||||||
|
conn.execute_batch(
|
||||||
|
"CREATE TABLE entries (feed_id TEXT NOT NULL, guid TEXT NOT NULL,
|
||||||
|
first_seen INTEGER NOT NULL, read INTEGER NOT NULL DEFAULT 0,
|
||||||
|
flagged INTEGER NOT NULL DEFAULT 0, position INTEGER NOT NULL DEFAULT 0,
|
||||||
|
PRIMARY KEY (feed_id, guid));
|
||||||
|
CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL UNIQUE COLLATE NOCASE,
|
||||||
|
pass_hash TEXT, is_admin INTEGER NOT NULL DEFAULT 0, created INTEGER NOT NULL);
|
||||||
|
CREATE TABLE subscriptions (
|
||||||
|
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||||
|
feed_id TEXT NOT NULL, created INTEGER NOT NULL, PRIMARY KEY (user_id, feed_id));
|
||||||
|
CREATE TABLE sessions (token TEXT PRIMARY KEY,
|
||||||
|
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||||
|
created INTEGER NOT NULL, seen INTEGER NOT NULL);
|
||||||
|
INSERT INTO users VALUES (1, 'ray', NULL, 1, 0);
|
||||||
|
INSERT INTO subscriptions VALUES (1, 'f', 0);
|
||||||
|
INSERT INTO sessions VALUES ('t', 1, 0, 0);",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
// The same order as open(): the schema leaves the old tables alone, migrate() fixes them.
|
||||||
|
conn.execute_batch(SCHEMA).unwrap();
|
||||||
|
migrate(&conn).unwrap();
|
||||||
|
let cols = |table: &str| -> Vec<String> {
|
||||||
|
conn.prepare(&format!("PRAGMA table_info({table})"))
|
||||||
|
.unwrap()
|
||||||
|
.query_map([], |r| r.get(1))
|
||||||
|
.unwrap()
|
||||||
|
.collect::<rusqlite::Result<_>>()
|
||||||
|
.unwrap()
|
||||||
|
};
|
||||||
|
for (table, gone) in [
|
||||||
|
("entries", &["read", "flagged", "position"][..]),
|
||||||
|
("subscriptions", &["created"][..]),
|
||||||
|
("sessions", &["created"][..]),
|
||||||
|
] {
|
||||||
|
let cols = cols(table);
|
||||||
|
assert!(!cols.iter().any(|c| gone.contains(&c.as_str())), "{table}: {cols:?}");
|
||||||
|
}
|
||||||
|
// users.created is not retired: it keeps what it held, and last_login joins it.
|
||||||
|
let users = cols("users");
|
||||||
|
assert!(users.iter().any(|c| c == "last_login"), "{users:?}");
|
||||||
|
assert_eq!(conn.query_row("SELECT created FROM users", [], |r| r.get::<_, i64>(0)).unwrap(), 0);
|
||||||
|
// And the rows come through it.
|
||||||
|
let kept: i64 = conn
|
||||||
|
.query_row("SELECT count(*) FROM subscriptions JOIN sessions USING (user_id)", [], |r| r.get(0))
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(kept, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_renamed_account_keeps_everything_but_its_name() {
|
||||||
|
let db = Db::memory().unwrap();
|
||||||
|
let ray = db.create_user("rays", None, true).unwrap();
|
||||||
|
db.create_user("sam", None, false).unwrap();
|
||||||
|
db.subscribe(ray, "f").unwrap();
|
||||||
|
db.rename_user(ray, "rays@sdf1.net").unwrap();
|
||||||
|
assert!(db.user_by_name("rays").unwrap().is_none());
|
||||||
|
let renamed = db.user_by_name("RAYS@sdf1.net").unwrap().unwrap();
|
||||||
|
assert_eq!((renamed.id, renamed.is_admin), (ray, true), "same account, still the admin");
|
||||||
|
assert_eq!(db.subscriptions_for(ray).unwrap().len(), 1, "and still subscribed");
|
||||||
|
assert!(db.rename_user(ray, "sam").is_err(), "a taken name is refused");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_account_knows_when_it_was_made_and_last_signed_in() {
|
||||||
|
let db = Db::memory().unwrap();
|
||||||
|
let id = db.create_user("ray", None, true).unwrap();
|
||||||
|
let get = || db.user_by_id(id).unwrap().unwrap();
|
||||||
|
assert!(get().created.is_some_and(|t| t > 0));
|
||||||
|
assert_eq!(get().last_login, None, "made, but never signed in");
|
||||||
|
db.signed_in(id).unwrap();
|
||||||
|
let first = get().last_login.unwrap();
|
||||||
|
// Within the hour, the proxy vouching again writes nothing; after it, it does.
|
||||||
|
db.exec_for_test(&format!("UPDATE users SET last_login = {} WHERE id = {id}", first - 60)).unwrap();
|
||||||
|
db.signed_in(id).unwrap();
|
||||||
|
assert_eq!(get().last_login, Some(first - 60));
|
||||||
|
db.exec_for_test(&format!("UPDATE users SET last_login = {} WHERE id = {id}", first - 7200)).unwrap();
|
||||||
|
db.signed_in(id).unwrap();
|
||||||
|
assert!(get().last_login.unwrap() >= first);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_first_admin_starts_with_the_catalogue_and_only_once() {
|
||||||
|
// Cutting this along with the dead read columns left the browser suite's admin with an
|
||||||
|
// empty sidebar: it is how a fresh install's first account gets config.toml's feeds.
|
||||||
|
let db = Db::memory().unwrap();
|
||||||
|
db.exec_for_test("INSERT INTO users (id, name, is_admin) VALUES (1,'admin',1);")
|
||||||
|
.unwrap();
|
||||||
|
let subs = || -> i64 {
|
||||||
|
db.conn.lock().unwrap().query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0)).unwrap()
|
||||||
|
};
|
||||||
|
let catalogue = ["a".to_string(), "b".to_string()];
|
||||||
|
assert_eq!(db.adopt_catalogue(1, &catalogue).unwrap(), 2);
|
||||||
|
assert_eq!(subs(), 2);
|
||||||
|
// Once anyone subscribes to anything it never runs again, so an unsubscribe sticks.
|
||||||
|
db.unsubscribe(1, "a").unwrap();
|
||||||
|
assert_eq!(db.adopt_catalogue(1, &catalogue).unwrap(), 0);
|
||||||
|
assert_eq!(subs(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn every_filter_works_with_and_without_a_search_term() {
|
fn every_filter_works_with_and_without_a_search_term() {
|
||||||
// Regression: the search clause used to be omitted when no term was given, while
|
// Regression: the search clause used to be omitted when no term was given, while
|
||||||
@@ -1496,7 +1645,7 @@ mod tests {
|
|||||||
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
|
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
|
||||||
(1,'f','b','u1','/tmp/b','done');
|
(1,'f','b','u1','/tmp/b','done');
|
||||||
-- Read and starred belong to a person now, so say which one.
|
-- Read and starred belong to a person now, so say which one.
|
||||||
INSERT INTO users (id, name, is_admin, created) VALUES (7,'reader',1,0);
|
INSERT INTO users (id, name, is_admin) VALUES (7,'reader',1);
|
||||||
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
|
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
|
||||||
(7,'f','b',1,0),
|
(7,'f','b',1,0),
|
||||||
(7,'f','c',1,1);",
|
(7,'f','c',1,1);",
|
||||||
@@ -1505,21 +1654,22 @@ mod tests {
|
|||||||
|
|
||||||
for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged] {
|
for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged] {
|
||||||
// Both paths must run without erroring, and agree with each other.
|
// Both paths must run without erroring, and agree with each other.
|
||||||
let rows = db.entries(7, "f", f, None, 0, 50).unwrap();
|
let order = order_sql("published", "desc");
|
||||||
let n = db.count_entries(7, "f", f, None).unwrap();
|
let rows = db.entries_in(7, Some("f"), f, None, 0, 50, &order).unwrap();
|
||||||
|
let n = db.count_in(7, Some("f"), f, None).unwrap();
|
||||||
assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page");
|
assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page");
|
||||||
|
|
||||||
let rows = db.entries(7, "f", f, Some("dive"), 0, 50).unwrap();
|
let rows = db.entries_in(7, Some("f"), f, Some("dive"), 0, 50, &order).unwrap();
|
||||||
let n = db.count_entries(7, "f", f, Some("dive")).unwrap();
|
let n = db.count_in(7, Some("f"), f, Some("dive")).unwrap();
|
||||||
assert_eq!(rows.len() as i64, n, "{f:?} with search disagrees");
|
assert_eq!(rows.len() as i64, n, "{f:?} with search disagrees");
|
||||||
}
|
}
|
||||||
|
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::All, None).unwrap(), 3);
|
assert_eq!(db.count_in(7, Some("f"), Filter::All, None).unwrap(), 3);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::Unread, None).unwrap(), 1);
|
assert_eq!(db.count_in(7, Some("f"), Filter::Unread, None).unwrap(), 1);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::Downloaded, None).unwrap(), 1);
|
assert_eq!(db.count_in(7, Some("f"), Filter::Downloaded, None).unwrap(), 1);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::Flagged, None).unwrap(), 1);
|
assert_eq!(db.count_in(7, Some("f"), Filter::Flagged, None).unwrap(), 1);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::All, Some("dive")).unwrap(), 2);
|
assert_eq!(db.count_in(7, Some("f"), Filter::All, Some("dive")).unwrap(), 2);
|
||||||
assert_eq!(db.count_entries(7, "f", Filter::All, Some("NOTES two")).unwrap(), 1,
|
assert_eq!(db.count_in(7, Some("f"), Filter::All, Some("NOTES two")).unwrap(), 1,
|
||||||
"search is case-insensitive and covers the description");
|
"search is case-insensitive and covers the description");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1561,6 +1711,61 @@ mod tests {
|
|||||||
assert_eq!(state(4), "done");
|
assert_eq!(state(4), "done");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_show_takes_over_what_its_creator_held() {
|
||||||
|
let db = Db::memory().unwrap();
|
||||||
|
db.exec_for_test(
|
||||||
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
|
||||||
|
INSERT INTO enclosures (id, feed_id, guid, url, state, path, last_error) VALUES
|
||||||
|
(1,'creator','a','u1','done','/x/a.mp3',NULL),
|
||||||
|
(2,'creator','b','u2','skipped',NULL,'explicit'),
|
||||||
|
(3,'creator','c','u3','skipped',NULL,'explicit'),
|
||||||
|
(4,'other','d','u4','skipped',NULL,'torrents disabled');
|
||||||
|
INSERT INTO entry_state (user_id, feed_id, guid, read) VALUES (1,'creator','a',1);",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
db.adopt("creator", "show", &[("a", "u1"), ("b", "u2"), ("d", "u4")]).unwrap();
|
||||||
|
{
|
||||||
|
let conn = db.conn.lock().unwrap();
|
||||||
|
let owner = |id: i64| -> String {
|
||||||
|
conn.query_row("SELECT feed_id FROM enclosures WHERE id = ?1", [id], |r| r.get(0)).unwrap()
|
||||||
|
};
|
||||||
|
assert_eq!(owner(1), "show", "a downloaded file moves with its item");
|
||||||
|
assert_eq!(owner(2), "show");
|
||||||
|
assert_eq!(owner(3), "creator", "this show does not list it");
|
||||||
|
assert_eq!(owner(4), "other", "only the parent's are taken");
|
||||||
|
let read: String = conn
|
||||||
|
.query_row("SELECT feed_id FROM entry_state WHERE user_id = 1 AND guid = 'a'", [], |r| r.get(0))
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(read, "show", "what you had read stays read");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only a filter's verdict can be overturned by a change of settings.
|
||||||
|
let skipped = db.skipped_by_filter("show").unwrap();
|
||||||
|
assert_eq!(skipped.get("u2").map(String::as_str), Some("explicit"));
|
||||||
|
assert_eq!(skipped.len(), 1);
|
||||||
|
assert!(db.skipped_by_filter("other").unwrap().is_empty(), "torrents disabled is not a filter");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_feed_in_a_group_follows_your_settings_on_the_group() {
|
||||||
|
let db = Db::memory().unwrap();
|
||||||
|
db.exec_for_test(
|
||||||
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
|
||||||
|
INSERT INTO subscriptions (user_id, feed_id, allow_explicit) VALUES
|
||||||
|
(1,'group',1),(1,'show',NULL),(2,'group',1),(2,'show',0);",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let explicit = |group| -> Vec<Option<bool>> {
|
||||||
|
let mut v: Vec<_> =
|
||||||
|
db.subscribers("show", group).unwrap().into_iter().map(|s| s.allow_explicit).collect();
|
||||||
|
v.sort();
|
||||||
|
v
|
||||||
|
};
|
||||||
|
assert_eq!(explicit(Some("group")), [Some(false), Some(true)], "ray inherits; sam's own choice on the show wins");
|
||||||
|
assert_eq!(explicit(None), [None, Some(false)], "outside a group nothing is inherited");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn enclosure_url_is_the_dedupe_key() {
|
fn enclosure_url_is_the_dedupe_key() {
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().unwrap();
|
||||||
|
|||||||
@@ -223,7 +223,7 @@ enum Sniffed {
|
|||||||
/// 2008 and so always answered 'data'.
|
/// 2008 and so always answered 'data'.
|
||||||
async fn sniff(path: &Path) -> Result<Sniffed> {
|
async fn sniff(path: &Path) -> Result<Sniffed> {
|
||||||
let head = read_head(path, 512).await?;
|
let head = read_head(path, 512).await?;
|
||||||
if infer::is(&head, "torrent") || head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
|
if head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
|
||||||
return Ok(Sniffed::Torrent);
|
return Ok(Sniffed::Torrent);
|
||||||
}
|
}
|
||||||
let text = String::from_utf8_lossy(&head);
|
let text = String::from_utf8_lossy(&head);
|
||||||
|
|||||||
175
src/feed.rs
175
src/feed.rs
@@ -117,6 +117,103 @@ pub fn opml_title(bytes: &[u8]) -> Option<String> {
|
|||||||
.filter(|t| !t.is_empty())
|
.filter(|t| !t.is_empty())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The token and show of a Patreon feed link, or None for any other URL.
|
||||||
|
///
|
||||||
|
/// Patreon gives each patron one token per creator. With no show it stands for the creator,
|
||||||
|
/// whose feed carries every show at once.
|
||||||
|
fn patreon_parts(url: &str) -> Option<(String, Option<String>)> {
|
||||||
|
let u = url::Url::parse(url).ok()?;
|
||||||
|
if !matches!(u.host_str()?, "patreon.com" | "www.patreon.com") || !u.path().starts_with("/rss") {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let param = |name: &str| u.query_pairs().find(|(k, _)| k == name).map(|(_, v)| v.into_owned());
|
||||||
|
Some((param("auth")?, param("show")))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A Patreon link naming a creator but no show.
|
||||||
|
pub fn is_patreon_creator(url: &str) -> bool {
|
||||||
|
matches!(patreon_parts(url), Some((_, None)))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What was typed into Add feed, as a URL. A bare Patreon token is taken as its creator's
|
||||||
|
/// feed, since the token alone says whose it is.
|
||||||
|
pub fn expand_input(input: &str) -> String {
|
||||||
|
let s = input.trim();
|
||||||
|
let token = s.len() >= 20 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_');
|
||||||
|
if token { format!("https://www.patreon.com/rss?auth={s}") } else { s.to_owned() }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether two URLs are the same feed. One Patreon show has several spellings -- by the
|
||||||
|
/// creator's name, by number, or with no creator at all -- and the token and show are what
|
||||||
|
/// identify it.
|
||||||
|
pub fn same_feed(a: &str, b: &str) -> bool {
|
||||||
|
a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A Patreon creator's name and shows, each show as (title, feed URL).
|
||||||
|
///
|
||||||
|
/// ponytail: Patreon's own web API, undocumented, asked without signing in. If it changes,
|
||||||
|
/// finding shows stops and the show feeds already found keep working. The documented API
|
||||||
|
/// needs an OAuth client per install and does not list shows.
|
||||||
|
pub async fn patreon_shows(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
url: &str,
|
||||||
|
) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||||
|
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
|
||||||
|
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
|
||||||
|
let mut resp = client.get(url).send().await.context("connecting")?;
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
|
||||||
|
}
|
||||||
|
let mut head = Vec::new();
|
||||||
|
while patreon_campaign(&head).is_none() && head.len() < 64 * 1024 {
|
||||||
|
let Some(chunk) = resp.chunk().await.context("reading the feed")? else { break };
|
||||||
|
head.extend_from_slice(&chunk);
|
||||||
|
}
|
||||||
|
let campaign = patreon_campaign(&head)
|
||||||
|
.ok_or_else(|| anyhow!("the Patreon feed does not say whose it is"))?;
|
||||||
|
|
||||||
|
let api = format!(
|
||||||
|
"https://www.patreon.com/api/campaigns/{campaign}\
|
||||||
|
?include=shows&fields%5Bcampaign%5D=name&fields%5Bcollection%5D=title"
|
||||||
|
);
|
||||||
|
let resp = client.get(api).send().await.context("asking Patreon for the shows")?;
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
return Err(anyhow!("Patreon would not list the shows: HTTP {}", resp.status()));
|
||||||
|
}
|
||||||
|
let (name, shows) = parse_patreon_shows(&resp.bytes().await.context("reading the shows")?)?;
|
||||||
|
Ok((name, shows.into_iter().map(|(id, title)| (title, format!("{url}&show={id}"))).collect()))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The campaign number in the start of a Patreon feed.
|
||||||
|
fn patreon_campaign(head: &[u8]) -> Option<String> {
|
||||||
|
let text = String::from_utf8_lossy(head);
|
||||||
|
text.match_indices("patreon.com/rss/").find_map(|(i, m)| {
|
||||||
|
let id: String = text[i + m.len()..].chars().take_while(char::is_ascii_digit).collect();
|
||||||
|
(!id.is_empty()).then_some(id)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A campaign's name and its shows as (id, title), from Patreon's JSON:API answer.
|
||||||
|
fn parse_patreon_shows(json: &[u8]) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||||
|
let v: serde_json::Value = serde_json::from_slice(json).context("Patreon's answer is not JSON")?;
|
||||||
|
// Missing is not the same as none. Read as no shows, the creator feed would be scanned as
|
||||||
|
// a plain feed, claim every show's files, and leave the shows empty once the list returned.
|
||||||
|
let ids = v["data"]["relationships"]["shows"]["data"]
|
||||||
|
.as_array()
|
||||||
|
.ok_or_else(|| anyhow!("Patreon's answer does not list the shows"))?;
|
||||||
|
let title = |id: &str| -> Option<String> {
|
||||||
|
let show = v["included"].as_array()?.iter().find(|x| x["type"] == "collection" && x["id"] == id)?;
|
||||||
|
show["attributes"]["title"].as_str().map(|t| t.trim().to_owned())
|
||||||
|
};
|
||||||
|
let shows = ids
|
||||||
|
.iter()
|
||||||
|
.filter_map(|s| s["id"].as_str())
|
||||||
|
.map(|id| (id.to_owned(), title(id).unwrap_or_else(|| format!("Show {id}"))))
|
||||||
|
.collect();
|
||||||
|
Ok((v["data"]["attributes"]["name"].as_str().map(str::to_owned), shows))
|
||||||
|
}
|
||||||
|
|
||||||
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
|
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
|
||||||
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
|
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
|
||||||
match rss::Channel::read_from(bytes) {
|
match rss::Channel::read_from(bytes) {
|
||||||
@@ -243,7 +340,7 @@ fn from_rss(ch: rss::Channel, bytes: &[u8]) -> ParsedFeed {
|
|||||||
link: non_empty(item.link()),
|
link: non_empty(item.link()),
|
||||||
published: item.pub_date().and_then(parse_date),
|
published: item.pub_date().and_then(parse_date),
|
||||||
// Content wins over description, as __getEntries preferred entry.content.
|
// Content wins over description, as __getEntries preferred entry.content.
|
||||||
description: non_empty(item.content()).or_else(|| non_empty(item.description())),
|
description: body(item.content(), item.description()),
|
||||||
categories: item
|
categories: item
|
||||||
.categories()
|
.categories()
|
||||||
.iter()
|
.iter()
|
||||||
@@ -309,11 +406,7 @@ fn from_atom(feed: atom_syndication::Feed) -> ParsedFeed {
|
|||||||
title: non_empty(Some(e.title().as_str())),
|
title: non_empty(Some(e.title().as_str())),
|
||||||
link: alt.map(str::to_owned),
|
link: alt.map(str::to_owned),
|
||||||
published: e.published().or(Some(e.updated())).map(|d| d.timestamp()),
|
published: e.published().or(Some(e.updated())).map(|d| d.timestamp()),
|
||||||
description: e
|
description: body(e.content().and_then(|c| c.value()), e.summary().map(|s| s.as_str())),
|
||||||
.content()
|
|
||||||
.and_then(|c| c.value())
|
|
||||||
.or_else(|| e.summary().map(|s| s.as_str()))
|
|
||||||
.map(str::to_owned),
|
|
||||||
categories: e.categories().iter().map(|c| c.term().to_owned()).collect(),
|
categories: e.categories().iter().map(|c| c.term().to_owned()).collect(),
|
||||||
explicit: false,
|
explicit: false,
|
||||||
image: None,
|
image: None,
|
||||||
@@ -358,6 +451,25 @@ fn non_empty(s: Option<&str>) -> Option<String> {
|
|||||||
s.map(str::trim).filter(|s| !s.is_empty()).map(str::to_owned)
|
s.map(str::trim).filter(|s| !s.is_empty()).map(str::to_owned)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// An item's show notes: its full body when that is whole, else its description.
|
||||||
|
///
|
||||||
|
/// libsyn served Daily Meditation Podcast's `content:encoded` cut at the `>` inside a class name
|
||||||
|
/// pasted from a web app (`[&:has([data-writing-block])>*]:pointer-events-auto`), so the body
|
||||||
|
/// began halfway through a tag and the page showed the rest of the tag as text. The same item's
|
||||||
|
/// `description` was whole. With no description to fall back on, a damaged body beats none.
|
||||||
|
fn body(content: Option<&str>, description: Option<&str>) -> Option<String> {
|
||||||
|
non_empty(content)
|
||||||
|
.filter(|c| !starts_mid_tag(c))
|
||||||
|
.or_else(|| non_empty(description))
|
||||||
|
.or_else(|| non_empty(content))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Text that closes an attribute list (`">`) before any tag has opened is the tail of a tag whose
|
||||||
|
/// start was cut off.
|
||||||
|
fn starts_mid_tag(html: &str) -> bool {
|
||||||
|
html[..html.find('<').unwrap_or(html.len())].contains("\">")
|
||||||
|
}
|
||||||
|
|
||||||
/// The picture to show beside an item, in order of how deliberate it is:
|
/// The picture to show beside an item, in order of how deliberate it is:
|
||||||
/// `itunes:image`, then Media RSS `media:thumbnail`, then a `media:content` that is an
|
/// `itunes:image`, then Media RSS `media:thumbnail`, then a `media:content` that is an
|
||||||
/// image, and finally an image enclosure -- which is how a blog's article picture arrives
|
/// image, and finally an image enclosure -- which is how a blog's article picture arrives
|
||||||
@@ -451,6 +563,18 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_body_cut_off_mid_tag_gives_way_to_the_description() {
|
||||||
|
// How libsyn served Daily Meditation Podcast #3477: content:encoded began inside a tag.
|
||||||
|
let cut = r#"*]:pointer-events-auto R6Vx5W_threadScrollVars" dir="auto" data-turn="assistant"> <p>What if</p>"#;
|
||||||
|
let whole = r#"<div class="[&:has([data-writing-block])>*]:pointer-events-auto"><p>What if</p></div>"#;
|
||||||
|
assert_eq!(body(Some(cut), Some(whole)).as_deref(), Some(whole));
|
||||||
|
assert_eq!(body(Some("<p>Notes</p>"), Some("Summary")).as_deref(), Some("<p>Notes</p>"), "a whole body wins");
|
||||||
|
assert_eq!(body(Some("Plain notes, no tags."), Some("Summary")).as_deref(), Some("Plain notes, no tags."));
|
||||||
|
assert_eq!(body(Some(cut), None).as_deref(), Some(cut), "a damaged body beats none");
|
||||||
|
assert_eq!(body(None, Some("Summary")).as_deref(), Some("Summary"));
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn feed_level_explicit_overrides_entries() {
|
fn feed_level_explicit_overrides_entries() {
|
||||||
let xml = br#"<?xml version="1.0"?>
|
let xml = br#"<?xml version="1.0"?>
|
||||||
@@ -550,6 +674,45 @@ mod tests {
|
|||||||
assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
|
assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_patreon_creator_is_a_list_of_its_shows() {
|
||||||
|
let tok = "AbCdEfGhIjKlMnOpQrStUvWxYz012_-9";
|
||||||
|
assert_eq!(expand_input(&format!(" {tok} ")), format!("https://www.patreon.com/rss?auth={tok}"));
|
||||||
|
assert_eq!(expand_input("https://example.com/rss"), "https://example.com/rss");
|
||||||
|
|
||||||
|
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss/glasscannon?auth={tok}")));
|
||||||
|
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss?auth={tok}")));
|
||||||
|
assert!(!is_patreon_creator(&format!("https://www.patreon.com/rss/x?auth={tok}&show=1")), "one show is a feed");
|
||||||
|
assert!(!is_patreon_creator(&format!("https://example.com/rss?auth={tok}")));
|
||||||
|
|
||||||
|
// The show you already have by name is the one a bare token would add by number.
|
||||||
|
assert!(same_feed(
|
||||||
|
&format!("https://www.patreon.com/rss/glasscannon?auth={tok}&show=2073588"),
|
||||||
|
&format!("https://www.patreon.com/rss?auth={tok}&show=2073588"),
|
||||||
|
));
|
||||||
|
assert!(!same_feed(
|
||||||
|
&format!("https://www.patreon.com/rss?auth={tok}&show=1"),
|
||||||
|
&format!("https://www.patreon.com/rss?auth={tok}&show=2"),
|
||||||
|
));
|
||||||
|
|
||||||
|
// The self link carries the campaign by number, whichever spelling was asked for.
|
||||||
|
let head = br#"<rss><channel><link>https://www.patreon.com/glasscannon</link>
|
||||||
|
<atom:link href="https://www.patreon.com/rss/369921?auth=t" rel="self"/>"#;
|
||||||
|
assert_eq!(patreon_campaign(head).as_deref(), Some("369921"));
|
||||||
|
assert_eq!(patreon_campaign(b"<rss><channel><title>T"), None);
|
||||||
|
|
||||||
|
let json = br#"{"data":{"id":"369921","type":"campaign","attributes":{"name":"The Glass Cannon Network"},
|
||||||
|
"relationships":{"shows":{"data":[{"id":"2073588","type":"collection"},{"id":"2073636","type":"collection"}]}}},
|
||||||
|
"included":[{"id":"2073588","type":"collection","attributes":{"title":"Get in the Trunk "}},
|
||||||
|
{"id":"2073636","type":"collection","attributes":{"title":"Shadowdark"}}]}"#;
|
||||||
|
let (name, shows) = parse_patreon_shows(json).unwrap();
|
||||||
|
assert_eq!(name.as_deref(), Some("The Glass Cannon Network"));
|
||||||
|
assert_eq!(shows, [("2073588".into(), "Get in the Trunk".into()), ("2073636".into(), "Shadowdark".into())]);
|
||||||
|
|
||||||
|
// An answer that stops naming the shows is an error, never "this creator has none".
|
||||||
|
assert!(parse_patreon_shows(br#"{"data":{"attributes":{"name":"X"}}}"#).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn an_item_may_carry_several_enclosures() {
|
fn an_item_may_carry_several_enclosures() {
|
||||||
// The rss crate keeps only one per item -- the last -- so these come from the XML.
|
// The rss crate keeps only one per item -- the last -- so these come from the XML.
|
||||||
|
|||||||
57
src/ipc.rs
57
src/ipc.rs
@@ -172,11 +172,18 @@ pub async fn daemon_is_live(path: &Path) -> bool {
|
|||||||
UnixStream::connect(path).await.is_ok()
|
UnixStream::connect(path).await.is_ok()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Answers `status` for the socket, without the worker. The worker runs one job at a time, and a
|
||||||
|
/// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon
|
||||||
|
/// dead. The answer goes to the client that asked and no one else: broadcast, it ended any
|
||||||
|
/// `ipx fetch` that was watching a scan, since `status` is a terminal event.
|
||||||
|
pub type StatusFn = std::sync::Arc<dyn Fn() -> Event + Send + Sync>;
|
||||||
|
|
||||||
/// Accepts connections, feeding commands to `cmds` and events from `events` back out.
|
/// Accepts connections, feeding commands to `cmds` and events from `events` back out.
|
||||||
pub async fn serve(
|
pub async fn serve(
|
||||||
path: PathBuf,
|
path: PathBuf,
|
||||||
events: broadcast::Sender<Event>,
|
events: broadcast::Sender<Event>,
|
||||||
cmds: mpsc::Sender<Command>,
|
cmds: mpsc::Sender<Command>,
|
||||||
|
status: StatusFn,
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
// A socket file left by a crashed daemon would block the bind; a live one was already
|
// A socket file left by a crashed daemon would block the bind; a live one was already
|
||||||
// rejected by the caller's daemon_is_live() check.
|
// rejected by the caller's daemon_is_live() check.
|
||||||
@@ -195,8 +202,9 @@ pub async fn serve(
|
|||||||
let (stream, _) = listener.accept().await?;
|
let (stream, _) = listener.accept().await?;
|
||||||
let rx = events.subscribe();
|
let rx = events.subscribe();
|
||||||
let cmds = cmds.clone();
|
let cmds = cmds.clone();
|
||||||
|
let status = status.clone();
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
if let Err(e) = handle(stream, rx, cmds).await {
|
if let Err(e) = handle(stream, rx, cmds, status).await {
|
||||||
tracing::debug!(error = %e, "client gone");
|
tracing::debug!(error = %e, "client gone");
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -207,12 +215,21 @@ async fn handle(
|
|||||||
stream: UnixStream,
|
stream: UnixStream,
|
||||||
mut rx: broadcast::Receiver<Event>,
|
mut rx: broadcast::Receiver<Event>,
|
||||||
cmds: mpsc::Sender<Command>,
|
cmds: mpsc::Sender<Command>,
|
||||||
|
status: StatusFn,
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
let (read, mut write) = stream.into_split();
|
let (read, mut write) = stream.into_split();
|
||||||
|
|
||||||
// Events out.
|
// Events out: everything broadcast, and the answers meant for this client alone.
|
||||||
|
let (reply, mut replies) = mpsc::channel::<Event>(4);
|
||||||
let writer = tokio::spawn(async move {
|
let writer = tokio::spawn(async move {
|
||||||
while let Ok(ev) = rx.recv().await {
|
loop {
|
||||||
|
let ev = tokio::select! {
|
||||||
|
Some(ev) = replies.recv() => ev,
|
||||||
|
got = rx.recv() => match got {
|
||||||
|
Ok(ev) => ev,
|
||||||
|
Err(_) => break,
|
||||||
|
},
|
||||||
|
};
|
||||||
let mut line = serde_json::to_string(&ev).unwrap_or_default();
|
let mut line = serde_json::to_string(&ev).unwrap_or_default();
|
||||||
line.push('\n');
|
line.push('\n');
|
||||||
if write.write_all(line.as_bytes()).await.is_err() {
|
if write.write_all(line.as_bytes()).await.is_err() {
|
||||||
@@ -229,6 +246,15 @@ async fn handle(
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
match serde_json::from_str::<Command>(line) {
|
match serde_json::from_str::<Command>(line) {
|
||||||
|
// Answered here, not queued behind whatever the worker is on: see StatusFn.
|
||||||
|
Ok(Command::Status) => {
|
||||||
|
tracing::info!(target: "ipx::io", "-> {line}");
|
||||||
|
let ev = status();
|
||||||
|
if let Ok(json) = serde_json::to_string(&ev) {
|
||||||
|
tracing::info!(target: "ipx::io", "<- {json}");
|
||||||
|
}
|
||||||
|
let _ = reply.send(ev).await;
|
||||||
|
}
|
||||||
Ok(cmd) => {
|
Ok(cmd) => {
|
||||||
if cmds.send(cmd).await.is_err() {
|
if cmds.send(cmd).await.is_err() {
|
||||||
break; // Worker is gone; so are we.
|
break; // Worker is gone; so are we.
|
||||||
@@ -329,4 +355,29 @@ mod tests {
|
|||||||
}
|
}
|
||||||
.is_terminal());
|
.is_terminal());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn status_is_answered_while_the_worker_is_busy() {
|
||||||
|
// The queue is full and nobody drains it, as when the worker is deep in a long download:
|
||||||
|
// anything sent to it would wait for ever.
|
||||||
|
let (cmds, _worker) = mpsc::channel::<Command>(1);
|
||||||
|
cmds.send(Command::Reap { dry_run: true }).await.unwrap();
|
||||||
|
let (events, _) = broadcast::channel::<Event>(8);
|
||||||
|
// Another client, watching a scan: it must not be handed someone else's answer, which
|
||||||
|
// would end its session.
|
||||||
|
let mut watcher = events.subscribe();
|
||||||
|
let status: StatusFn = std::sync::Arc::new(|| Event::Status { feeds: 1, pending: 2, downloaded: 3 });
|
||||||
|
let (client, server) = UnixStream::pair().unwrap();
|
||||||
|
tokio::spawn(handle(server, events.subscribe(), cmds, status));
|
||||||
|
|
||||||
|
let (read, mut write) = client.into_split();
|
||||||
|
write.write_all(b"{\"cmd\":\"status\"}\n").await.unwrap();
|
||||||
|
let line = tokio::time::timeout(std::time::Duration::from_secs(2), BufReader::new(read).lines().next_line())
|
||||||
|
.await
|
||||||
|
.expect("status waited behind the worker")
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert!(line.contains(r#""ev":"status""#), "{line}");
|
||||||
|
assert!(watcher.try_recv().is_err(), "the answer went to every client, not just the one asking");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -111,18 +111,11 @@ impl Visit for Collect {
|
|||||||
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
||||||
self.add(field, format!("{value:?}"));
|
self.add(field, format!("{value:?}"));
|
||||||
}
|
}
|
||||||
|
// Numbers and bools reach record_debug through the trait's defaults, which prints them the
|
||||||
|
// same way. A string would print quoted there, hence its own method.
|
||||||
fn record_str(&mut self, field: &Field, value: &str) {
|
fn record_str(&mut self, field: &Field, value: &str) {
|
||||||
self.add(field, value.to_owned());
|
self.add(field, value.to_owned());
|
||||||
}
|
}
|
||||||
fn record_i64(&mut self, field: &Field, value: i64) {
|
|
||||||
self.add(field, value.to_string());
|
|
||||||
}
|
|
||||||
fn record_u64(&mut self, field: &Field, value: u64) {
|
|
||||||
self.add(field, value.to_string());
|
|
||||||
}
|
|
||||||
fn record_bool(&mut self, field: &Field, value: bool) {
|
|
||||||
self.add(field, value.to_string());
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
|
|||||||
202
src/main.rs
202
src/main.rs
@@ -99,6 +99,9 @@ enum UserCmd {
|
|||||||
Passwd { name: String },
|
Passwd { name: String },
|
||||||
/// Delete an account and everything it knows: its subscriptions and read state
|
/// Delete an account and everything it knows: its subscriptions and read state
|
||||||
Rm { name: String },
|
Rm { name: String },
|
||||||
|
/// Rename an account, keeping its feeds, read state and admin rights. This is how an
|
||||||
|
/// account made before the proxy takes the name the proxy signs it in as
|
||||||
|
Rename { name: String, new_name: String },
|
||||||
}
|
}
|
||||||
|
|
||||||
/// What a brand new database starts with, so there is always a way in. Announced loudly
|
/// What a brand new database starts with, so there is always a way in. Announced loudly
|
||||||
@@ -273,11 +276,16 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
|||||||
println!("no accounts yet: ipx user add <name>");
|
println!("no accounts yet: ipx user add <name>");
|
||||||
}
|
}
|
||||||
for u in users {
|
for u in users {
|
||||||
|
let added = u
|
||||||
|
.created
|
||||||
|
.and_then(|t| chrono::DateTime::from_timestamp(t, 0))
|
||||||
|
.map_or("?".into(), |d| d.format("%Y-%m-%d").to_string());
|
||||||
|
let seen = u.last_login.map_or("never signed in".into(), |t| format!("signed in {}", ago(Some(t))));
|
||||||
println!(
|
println!(
|
||||||
"{:<20} {:<8} {}",
|
"{:<20} {:<6} {:<11} added {added} {seen}",
|
||||||
u.name,
|
u.name,
|
||||||
if u.is_admin { "admin" } else { "" },
|
if u.is_admin { "admin" } else { "" },
|
||||||
if u.pass_hash.is_some() { "password" } else { "proxy only" }
|
if u.pass_hash.is_some() { "password" } else { "proxy only" },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
@@ -292,6 +300,22 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
|||||||
println!("password changed for {name}");
|
println!("password changed for {name}");
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
UserCmd::Rename { name, new_name } => {
|
||||||
|
let name = name.trim().to_ascii_lowercase();
|
||||||
|
// The same rules as a name the proxy vouches for, or the proxy would never find it.
|
||||||
|
let new_name = crate::auth::name_from_header(&new_name)
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("not a usable name: no commas, semicolons or line breaks"))?;
|
||||||
|
let user = ctx
|
||||||
|
.db
|
||||||
|
.user_by_name(&name)?
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
|
||||||
|
if ctx.db.user_by_name(&new_name)?.is_some() {
|
||||||
|
anyhow::bail!("{new_name} already exists");
|
||||||
|
}
|
||||||
|
ctx.db.rename_user(user.id, &new_name)?;
|
||||||
|
println!("renamed {name} to {new_name}");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
UserCmd::Rm { name } => {
|
UserCmd::Rm { name } => {
|
||||||
let name = name.trim().to_ascii_lowercase();
|
let name = name.trim().to_ascii_lowercase();
|
||||||
let user = ctx
|
let user = ctx
|
||||||
@@ -315,14 +339,24 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
|
|||||||
Cmd::Reap { dry_run } => reap(ctx, dry_run, true),
|
Cmd::Reap { dry_run } => reap(ctx, dry_run, true),
|
||||||
Cmd::Download { enclosure } => download_one(ctx, enclosure).await,
|
Cmd::Download { enclosure } => download_one(ctx, enclosure).await,
|
||||||
Cmd::Status => {
|
Cmd::Status => {
|
||||||
let (pending, downloaded) = ctx.db.counts()?;
|
ctx.out.emit(status(ctx));
|
||||||
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0);
|
|
||||||
ctx.out.emit(Event::Status { feeds, pending, downloaded });
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The counts `ipx status` prints. A running daemon's socket answers with this directly rather
|
||||||
|
/// than through the job queue.
|
||||||
|
fn status(ctx: &Ctx) -> Event {
|
||||||
|
match ctx.db.counts() {
|
||||||
|
Ok((pending, downloaded)) => {
|
||||||
|
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0);
|
||||||
|
Event::Status { feeds, pending, downloaded }
|
||||||
|
}
|
||||||
|
Err(e) => Event::Error { msg: format!("{e:#}") },
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async fn daemon(
|
async fn daemon(
|
||||||
ctx: Arc<Ctx>,
|
ctx: Arc<Ctx>,
|
||||||
config_path: PathBuf,
|
config_path: PathBuf,
|
||||||
@@ -334,8 +368,7 @@ async fn daemon(
|
|||||||
anyhow::bail!("a daemon is already listening on {}", socket.display());
|
anyhow::bail!("a daemon is already listening on {}", socket.display());
|
||||||
}
|
}
|
||||||
|
|
||||||
// A database with nobody in it cannot be signed into, and an install that predates
|
// A database with nobody in it cannot be signed into.
|
||||||
// accounts still has to serve its owner. Both get the same starting point.
|
|
||||||
if ctx.db.users()?.is_empty() {
|
if ctx.db.users()?.is_empty() {
|
||||||
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
|
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
|
||||||
tracing::warn!(
|
tracing::warn!(
|
||||||
@@ -344,22 +377,15 @@ async fn daemon(
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// A library that predates accounts belongs to whoever was using it: the admin.
|
|
||||||
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
|
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
|
||||||
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
|
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
|
||||||
match ctx.db.adopt_existing_library(admin.id, &catalogue) {
|
match ctx.db.adopt_catalogue(admin.id, &catalogue) {
|
||||||
Ok(0) => {}
|
Ok(0) => {}
|
||||||
Ok(n) => tracing::info!(user = %admin.name, entries = n, "adopted the existing library"),
|
Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
|
||||||
Err(e) => tracing::error!(error = %e, "could not adopt the existing library"),
|
Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
match migrate_opml_children(&ctx) {
|
|
||||||
Ok(n) if n > 0 => tracing::info!(count = n, "moved OPML feeds out of config.toml into the database"),
|
|
||||||
Ok(_) => {}
|
|
||||||
Err(e) => tracing::warn!(error = ?e, "could not tidy OPML feeds out of the config"),
|
|
||||||
}
|
|
||||||
|
|
||||||
match ctx.db.requeue_interrupted() {
|
match ctx.db.requeue_interrupted() {
|
||||||
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
|
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
|
||||||
Ok(_) => {}
|
Ok(_) => {}
|
||||||
@@ -369,7 +395,12 @@ async fn daemon(
|
|||||||
let (tx_cmd, mut rx_cmd) = mpsc::channel::<Cmd>(64);
|
let (tx_cmd, mut rx_cmd) = mpsc::channel::<Cmd>(64);
|
||||||
|
|
||||||
let web = start_web(&ctx, &config_path, web_addr, &tx_cmd, &events).await?;
|
let web = start_web(&ctx, &config_path, web_addr, &tx_cmd, &events).await?;
|
||||||
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd));
|
// status is answered by the socket itself; everything else waits its turn in the queue.
|
||||||
|
let answer: ipc::StatusFn = {
|
||||||
|
let ctx = ctx.clone();
|
||||||
|
Arc::new(move || status(&ctx))
|
||||||
|
};
|
||||||
|
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer));
|
||||||
|
|
||||||
// One command at a time: the queue is what keeps two scans from overlapping.
|
// One command at a time: the queue is what keeps two scans from overlapping.
|
||||||
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
|
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
|
||||||
@@ -468,7 +499,7 @@ async fn start_web(
|
|||||||
let mut fresh = (*cfg).clone();
|
let mut fresh = (*cfg).clone();
|
||||||
fresh.web.enabled = true;
|
fresh.web.enabled = true;
|
||||||
fresh.web.bind = bind.clone();
|
fresh.web.bind = bind.clone();
|
||||||
fresh.web.token = web::generate_token();
|
fresh.web.token = crate::auth::new_session_token();
|
||||||
fresh.save(config_path)?;
|
fresh.save(config_path)?;
|
||||||
ctx.reload_cfg(config_path)?;
|
ctx.reload_cfg(config_path)?;
|
||||||
println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token);
|
println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token);
|
||||||
@@ -517,8 +548,9 @@ async fn add(
|
|||||||
keywords: Vec<String>,
|
keywords: Vec<String>,
|
||||||
) -> Result<()> {
|
) -> Result<()> {
|
||||||
let mut cfg = (*ctx.cfg()).clone();
|
let mut cfg = (*ctx.cfg()).clone();
|
||||||
|
let url = &feed::expand_input(url);
|
||||||
// Includes feeds derived from an OPML, or the same show could be added twice.
|
// Includes feeds derived from an OPML, or the same show could be added twice.
|
||||||
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| s.cfg.url == url) {
|
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
|
||||||
anyhow::bail!("already subscribed as {:?}", existing.id);
|
anyhow::bail!("already subscribed as {:?}", existing.id);
|
||||||
}
|
}
|
||||||
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
|
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
|
||||||
@@ -569,10 +601,17 @@ pub async fn add_one(
|
|||||||
|
|
||||||
// Slugs must be unique across derived feeds too, or a new feed can collide with one
|
// Slugs must be unique across derived feeds too, or a new feed can collide with one
|
||||||
// an OPML already introduced.
|
// an OPML already introduced.
|
||||||
let taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
|
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.map(|s| (s.id, s.cfg))
|
.map(|s| (s.id, s.cfg))
|
||||||
.collect();
|
.collect();
|
||||||
|
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
|
||||||
|
// it gets its history back, and a different feed does not inherit someone else's.
|
||||||
|
for (id, other) in ctx.db.feed_urls()? {
|
||||||
|
if !feed::same_feed(&other, url) {
|
||||||
|
taken.entry(id).or_insert_with(|| probe.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
let id = config::unique_slug(&title, &taken);
|
let id = config::unique_slug(&title, &taken);
|
||||||
cfg.feeds.insert(id.clone(), probe);
|
cfg.feeds.insert(id.clone(), probe);
|
||||||
Ok(id)
|
Ok(id)
|
||||||
@@ -807,7 +846,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
|
|||||||
ctx.out.emit(Event::FeedSkip {
|
ctx.out.emit(Event::FeedSkip {
|
||||||
feed: id.clone(),
|
feed: id.clone(),
|
||||||
reason: format!(
|
reason: format!(
|
||||||
"OPML: {total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
|
"{total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
|
||||||
added.len()
|
added.len()
|
||||||
),
|
),
|
||||||
});
|
});
|
||||||
@@ -908,36 +947,6 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
|
|||||||
Ok(out)
|
Ok(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Moves OPML children that older versions wrote into config.toml over to the database.
|
|
||||||
/// They were never yours to edit, and 80-odd of them made the file unreadable.
|
|
||||||
fn migrate_opml_children(ctx: &Ctx) -> Result<usize> {
|
|
||||||
let cfg = (*ctx.cfg()).clone();
|
|
||||||
let children: Vec<(String, config::Feed)> = cfg
|
|
||||||
.feeds
|
|
||||||
.iter()
|
|
||||||
.filter(|(_, f)| f.group.is_some())
|
|
||||||
.map(|(id, f)| (id.clone(), f.clone()))
|
|
||||||
.collect();
|
|
||||||
if children.is_empty() {
|
|
||||||
return Ok(0);
|
|
||||||
}
|
|
||||||
let mut fresh = cfg.clone();
|
|
||||||
for (id, f) in &children {
|
|
||||||
let group = f.group.clone().unwrap_or_default();
|
|
||||||
let title = ctx
|
|
||||||
.db
|
|
||||||
.feed_summary(id)
|
|
||||||
.ok()
|
|
||||||
.and_then(|s| s.title)
|
|
||||||
.unwrap_or_else(|| id.clone());
|
|
||||||
ctx.db.upsert_managed(id, &f.url, &title, &group)?;
|
|
||||||
fresh.feeds.remove(id);
|
|
||||||
}
|
|
||||||
fresh.save(&ctx.config_path)?;
|
|
||||||
ctx.reload_cfg(&ctx.config_path)?;
|
|
||||||
Ok(children.len())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Seconds to wait before re-checking a feed.
|
/// Seconds to wait before re-checking a feed.
|
||||||
///
|
///
|
||||||
/// A per-feed schedule is an explicit instruction and wins outright. Without one, the
|
/// A per-feed schedule is an explicit instruction and wins outright. Without one, the
|
||||||
@@ -963,7 +972,7 @@ struct Scan {
|
|||||||
enum Outcome {
|
enum Outcome {
|
||||||
NotModified,
|
NotModified,
|
||||||
Feed(Scan),
|
Feed(Scan),
|
||||||
/// The URL served an OPML document, so it is a subscription list rather than a feed.
|
/// The URL is a list of feeds rather than a feed: an OPML, or a Patreon creator's shows.
|
||||||
Opml { added: Vec<String>, removed: usize, kept: usize, total: usize },
|
Opml { added: Vec<String>, removed: usize, kept: usize, total: usize },
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -973,6 +982,31 @@ async fn scan_one(
|
|||||||
feed_cfg: &config::Feed,
|
feed_cfg: &config::Feed,
|
||||||
state: &db::HttpState,
|
state: &db::HttpState,
|
||||||
) -> Result<Outcome> {
|
) -> Result<Outcome> {
|
||||||
|
// A Patreon creator with more than one show is a list of feeds, like an OPML.
|
||||||
|
if feed::is_patreon_creator(&feed_cfg.url) {
|
||||||
|
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
|
||||||
|
Ok((name, shows)) if shows.len() > 1 => {
|
||||||
|
ctx.db.touch_feed(id, &feed_cfg.url)?;
|
||||||
|
if let Some(name) = name {
|
||||||
|
ctx.db.set_title(id, &name)?;
|
||||||
|
}
|
||||||
|
// Read as one feed before it was split, it listed every show's items in one
|
||||||
|
// heap. The items go; its files and read state move to each show as the show
|
||||||
|
// lists them (`Db::adopt`), so no show comes up empty for want of a URL.
|
||||||
|
ctx.db.clear_entries(id)?;
|
||||||
|
return sync_group(ctx, id, feed_cfg, &shows).await;
|
||||||
|
}
|
||||||
|
Ok(_) => {} // One show: the creator's feed is that show.
|
||||||
|
// Already split: keep the shows it has rather than read the creator as one heap.
|
||||||
|
Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e),
|
||||||
|
Err(e) => tracing::warn!(
|
||||||
|
feed = id,
|
||||||
|
error = %format!("{e:#}"),
|
||||||
|
"could not list the Patreon shows; reading it as one feed"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
let mut fetched = feed::fetch(
|
let mut fetched = feed::fetch(
|
||||||
&ctx.client,
|
&ctx.client,
|
||||||
feed_cfg,
|
feed_cfg,
|
||||||
@@ -1018,19 +1052,38 @@ async fn scan_one(
|
|||||||
)?;
|
)?;
|
||||||
|
|
||||||
let policy = policy_for(ctx, id, feed_cfg)?;
|
let policy = policy_for(ctx, id, feed_cfg)?;
|
||||||
|
if let Some(parent) = &feed_cfg.group {
|
||||||
|
let listed: Vec<(&str, &str)> = parsed
|
||||||
|
.entries
|
||||||
|
.iter()
|
||||||
|
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
|
||||||
|
.collect();
|
||||||
|
ctx.db.adopt(parent, id, &listed)?;
|
||||||
|
}
|
||||||
|
// Verdicts are recorded in `state`, so the download queue below is just "everything still
|
||||||
|
// pending". A filter's verdict is looked at again on every scan, though: made once, at
|
||||||
|
// discovery, it outlived the setting behind it, and allowing explicit items afterwards
|
||||||
|
// changed nothing however often the feed was scanned.
|
||||||
|
let skipped = ctx.db.skipped_by_filter(id)?;
|
||||||
let mut scan = Scan::default();
|
let mut scan = Scan::default();
|
||||||
for entry in &parsed.entries {
|
for entry in &parsed.entries {
|
||||||
if ctx.db.record_entry(id, entry)? {
|
if ctx.db.record_entry(id, entry)? {
|
||||||
scan.new_entries += 1;
|
scan.new_entries += 1;
|
||||||
}
|
}
|
||||||
for enc in &entry.enclosures {
|
for enc in &entry.enclosures {
|
||||||
if !ctx.db.record_enclosure(id, &entry.guid, enc)? {
|
let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? {
|
||||||
continue; // Seen before: downloaded, skipped or deliberately reaped.
|
None
|
||||||
|
} else if let Some(reason) = skipped.get(&enc.url) {
|
||||||
|
Some(reason.as_str())
|
||||||
|
} else {
|
||||||
|
continue; // Settled: queued, downloaded, reaped, or another feed's file.
|
||||||
|
};
|
||||||
|
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
|
||||||
|
if now != was {
|
||||||
|
match now {
|
||||||
|
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?,
|
||||||
|
None => ctx.db.mark_enclosure(&enc.url, "pending", None)?,
|
||||||
}
|
}
|
||||||
// Filters run once, at discovery, and are recorded in `state`. The download
|
|
||||||
// queue below is then just "everything still pending".
|
|
||||||
if let Some(reason) = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc) {
|
|
||||||
ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1114,12 +1167,6 @@ async fn scan_one(
|
|||||||
Ok(Outcome::Feed(scan))
|
Ok(Outcome::Feed(scan))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Brings the feed list in step with a subscribed OPML.
|
|
||||||
///
|
|
||||||
/// New entries are added under the OPML's group and folder. An entry that has gone from
|
|
||||||
/// the OPML is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
|
|
||||||
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
|
|
||||||
/// the UI to explain them.
|
|
||||||
async fn sync_opml(
|
async fn sync_opml(
|
||||||
ctx: &Arc<Ctx>,
|
ctx: &Arc<Ctx>,
|
||||||
parent_id: &str,
|
parent_id: &str,
|
||||||
@@ -1130,25 +1177,44 @@ async fn sync_opml(
|
|||||||
if let Some(title) = feed::opml_title(bytes) {
|
if let Some(title) = feed::opml_title(bytes) {
|
||||||
ctx.db.set_title(parent_id, &title)?;
|
ctx.db.set_title(parent_id, &title)?;
|
||||||
}
|
}
|
||||||
|
sync_group(ctx, parent_id, parent, &listed).await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Brings the feed list in step with a list of feeds: a subscribed OPML, or a Patreon
|
||||||
|
/// creator's shows.
|
||||||
|
///
|
||||||
|
/// New entries are added under the list's group and folder. An entry that has gone from
|
||||||
|
/// the list is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
|
||||||
|
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
|
||||||
|
/// the UI to explain them.
|
||||||
|
async fn sync_group(
|
||||||
|
ctx: &Arc<Ctx>,
|
||||||
|
parent_id: &str,
|
||||||
|
parent: &config::Feed,
|
||||||
|
listed: &[(String, String)],
|
||||||
|
) -> Result<Outcome> {
|
||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
let existing = ctx.db.managed_feeds()?;
|
let existing = ctx.db.managed_feeds()?;
|
||||||
let mut added = vec![];
|
let mut added = vec![];
|
||||||
|
|
||||||
for (title, url) in &listed {
|
for (title, url) in listed {
|
||||||
// Already known, whether derived or promoted into the config.
|
// Already known, whether derived or promoted into the config.
|
||||||
if let Some(m) = existing.iter().find(|m| &m.url == url) {
|
if let Some(m) = existing.iter().find(|m| &m.url == url) {
|
||||||
ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
|
ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if cfg.feeds.values().any(|f| &f.url == url) {
|
// A Patreon show you added by hand may be spelled differently from the one listed.
|
||||||
|
if cfg.feeds.values().any(|f| feed::same_feed(&f.url, url)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
// A removed feed keeps its rows, so its id is only free again for the same feed.
|
||||||
|
let known = ctx.db.feed_urls()?;
|
||||||
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
|
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
|
||||||
.feeds
|
.feeds
|
||||||
.keys()
|
.keys()
|
||||||
.chain(existing.iter().map(|m| &m.id))
|
.chain(existing.iter().map(|m| &m.id))
|
||||||
.chain(added.iter())
|
.chain(added.iter())
|
||||||
|
.chain(known.iter().filter(|(_, u)| !feed::same_feed(u, url)).map(|(id, _)| id))
|
||||||
.map(|id| (id.clone(), parent.clone()))
|
.map(|id| (id.clone(), parent.clone()))
|
||||||
.collect();
|
.collect();
|
||||||
let id = config::unique_slug(title, &taken);
|
let id = config::unique_slug(title, &taken);
|
||||||
@@ -1255,7 +1321,7 @@ pub struct Policy {
|
|||||||
|
|
||||||
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
|
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
|
||||||
let global = ctx.cfg().general.max_new_per_check;
|
let global = ctx.cfg().general.max_new_per_check;
|
||||||
Ok(merge_policy(&ctx.db.subscribers(id)?, feed_cfg, global))
|
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
|
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
|
||||||
|
|||||||
@@ -154,8 +154,8 @@ mod tests {
|
|||||||
// One file serves both subscribers, so it takes both of them to release it.
|
// One file serves both subscribers, so it takes both of them to release it.
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().unwrap();
|
||||||
db.exec_for_test(
|
db.exec_for_test(
|
||||||
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0);
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
|
||||||
INSERT INTO subscriptions (user_id, feed_id, created) VALUES (1,'f',0),(2,'f',0);
|
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f');
|
||||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
||||||
('f', 'keep', 0),
|
('f', 'keep', 0),
|
||||||
('f', 'half', 0),
|
('f', 'half', 0),
|
||||||
@@ -189,7 +189,7 @@ mod tests {
|
|||||||
fn prune_keeps_entries_that_still_have_a_file() {
|
fn prune_keeps_entries_that_still_have_a_file() {
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().unwrap();
|
||||||
db.exec_for_test(
|
db.exec_for_test(
|
||||||
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0);
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
|
||||||
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',1);
|
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',1);
|
||||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
||||||
('f', 'has-file', 100),
|
('f', 'has-file', 100),
|
||||||
|
|||||||
221
src/web.rs
221
src/web.rs
@@ -12,9 +12,7 @@ use axum::{
|
|||||||
},
|
},
|
||||||
routing::{delete, get, patch, post},
|
routing::{delete, get, patch, post},
|
||||||
};
|
};
|
||||||
use futures_util::StreamExt;
|
|
||||||
use serde::Deserialize;
|
use serde::Deserialize;
|
||||||
use tokio_stream::wrappers::BroadcastStream;
|
|
||||||
use tower::ServiceExt;
|
use tower::ServiceExt;
|
||||||
use tower_http::services::ServeFile;
|
use tower_http::services::ServeFile;
|
||||||
use serde::Serialize;
|
use serde::Serialize;
|
||||||
@@ -35,28 +33,6 @@ pub struct WebState {
|
|||||||
pub events: broadcast::Sender<Event>,
|
pub events: broadcast::Sender<Event>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A 32-hex-character shared secret, generated when config.toml has none.
|
|
||||||
///
|
|
||||||
/// ponytail: /dev/urandom rather than a CSPRNG crate -- 16 bytes, once, on a Unix-only
|
|
||||||
/// binary. Falls back to the clock only if urandom is somehow unreadable, which would be a
|
|
||||||
/// weak token, so that case is logged loudly.
|
|
||||||
pub fn generate_token() -> String {
|
|
||||||
use std::io::Read;
|
|
||||||
let mut bytes = [0u8; 16];
|
|
||||||
match std::fs::File::open("/dev/urandom").and_then(|mut f| f.read_exact(&mut bytes)) {
|
|
||||||
Ok(()) => {}
|
|
||||||
Err(e) => {
|
|
||||||
tracing::error!(error = %e, "could not read /dev/urandom; token is NOT secure");
|
|
||||||
let n = std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.map(|d| d.as_nanos() as u64)
|
|
||||||
.unwrap_or(0);
|
|
||||||
bytes[..8].copy_from_slice(&n.to_le_bytes());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
bytes.iter().map(|b| format!("{b:02x}")).collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn router(state: WebState) -> Router {
|
pub fn router(state: WebState) -> Router {
|
||||||
Router::new()
|
Router::new()
|
||||||
.route("/", get(index))
|
.route("/", get(index))
|
||||||
@@ -88,6 +64,7 @@ pub fn router(state: WebState) -> Router {
|
|||||||
// Signing in cannot require being signed in, so these sit outside the auth layer.
|
// Signing in cannot require being signed in, so these sit outside the auth layer.
|
||||||
.route("/login", get(login_page))
|
.route("/login", get(login_page))
|
||||||
.route("/api/login", post(login))
|
.route("/api/login", post(login))
|
||||||
|
.route("/icon.png", get(icon))
|
||||||
.layer(middleware::from_fn(access_log))
|
.layer(middleware::from_fn(access_log))
|
||||||
.with_state(state)
|
.with_state(state)
|
||||||
}
|
}
|
||||||
@@ -114,23 +91,8 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
let cfg = state.ctx.cfg();
|
let cfg = state.ctx.cfg();
|
||||||
let token = cfg.web.token.clone();
|
let token = cfg.web.token.clone();
|
||||||
|
|
||||||
let peer = req
|
// 1. A header, but only from a hop we were told to believe.
|
||||||
.extensions()
|
let vouched = vouched_name(&cfg, &req);
|
||||||
.get::<axum::extract::ConnectInfo<std::net::SocketAddr>>()
|
|
||||||
.map(|c| c.0.ip().to_string())
|
|
||||||
.unwrap_or_default();
|
|
||||||
|
|
||||||
// 1. A header, but only from a hop we were told to believe. Anyone able to reach the
|
|
||||||
// port could otherwise send it and be whoever they liked.
|
|
||||||
let vouched = (!cfg.web.trusted_header.is_empty()
|
|
||||||
&& cfg.web.trusted_proxies.iter().any(|p| p == &peer))
|
|
||||||
.then(|| {
|
|
||||||
req.headers()
|
|
||||||
.get(&cfg.web.trusted_header)
|
|
||||||
.and_then(|v| v.to_str().ok())
|
|
||||||
.and_then(crate::auth::name_from_header)
|
|
||||||
})
|
|
||||||
.flatten();
|
|
||||||
|
|
||||||
let mut set_cookie: Option<String> = None;
|
let mut set_cookie: Option<String> = None;
|
||||||
let mut user = None;
|
let mut user = None;
|
||||||
@@ -156,7 +118,13 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
None
|
None
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
// Every request comes vouched for; signed_in keeps one an hour. Failing to note the time
|
||||||
|
// must not turn anyone away, so its error goes unanswered.
|
||||||
|
if let Some(u) = &user {
|
||||||
|
let _ = state.ctx.db.signed_in(u.id);
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
let by_proxy = user.is_some();
|
||||||
|
|
||||||
// 2. A session cookie from signing in here.
|
// 2. A session cookie from signing in here.
|
||||||
if user.is_none() {
|
if user.is_none() {
|
||||||
@@ -180,6 +148,10 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
if supplied.is_some_and(|t| constant_time_eq(&t, &token)) {
|
if supplied.is_some_and(|t| constant_time_eq(&t, &token)) {
|
||||||
user = admin_user(&state);
|
user = admin_user(&state);
|
||||||
if from_query.is_some() {
|
if from_query.is_some() {
|
||||||
|
// The token link is a sign-in; the cookie it leaves behind is not one each time.
|
||||||
|
if let Some(u) = &user {
|
||||||
|
let _ = state.ctx.db.signed_in(u.id);
|
||||||
|
}
|
||||||
set_cookie = Some(format!(
|
set_cookie = Some(format!(
|
||||||
"{COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000"
|
"{COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000"
|
||||||
));
|
));
|
||||||
@@ -203,6 +175,7 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
};
|
};
|
||||||
|
|
||||||
req.extensions_mut().insert(user);
|
req.extensions_mut().insert(user);
|
||||||
|
req.extensions_mut().insert(Proxied(by_proxy));
|
||||||
let mut resp = next.run(req).await;
|
let mut resp = next.run(req).await;
|
||||||
if let Some(c) = set_cookie {
|
if let Some(c) = set_cookie {
|
||||||
if let Ok(v) = header::HeaderValue::from_str(&c) {
|
if let Ok(v) = header::HeaderValue::from_str(&c) {
|
||||||
@@ -214,6 +187,29 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
|
|
||||||
const SESSION_COOKIE: &str = "ipx_session";
|
const SESSION_COOKIE: &str = "ipx_session";
|
||||||
|
|
||||||
|
/// Whether the proxy signed this request in, rather than a session or the token: signing out
|
||||||
|
/// has to go through the proxy then, or its next request signs the person straight back in.
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
struct Proxied(bool);
|
||||||
|
|
||||||
|
/// The name the proxy vouches for, when this request came from one of `trusted_proxies` and
|
||||||
|
/// carries `trusted_header`. Anyone able to reach the port could otherwise send the header and
|
||||||
|
/// be whoever they liked.
|
||||||
|
fn vouched_name(cfg: &crate::config::Config, req: &Request) -> Option<String> {
|
||||||
|
let peer = req
|
||||||
|
.extensions()
|
||||||
|
.get::<axum::extract::ConnectInfo<std::net::SocketAddr>>()
|
||||||
|
.map(|c| c.0.ip().to_string())
|
||||||
|
.unwrap_or_default();
|
||||||
|
if cfg.web.trusted_header.is_empty() || !cfg.web.trusted_proxies.iter().any(|p| p == &peer) {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
req.headers()
|
||||||
|
.get(&cfg.web.trusted_header)
|
||||||
|
.and_then(|v| v.to_str().ok())
|
||||||
|
.and_then(crate::auth::name_from_header)
|
||||||
|
}
|
||||||
|
|
||||||
/// Handlers take `User` to say they need one; the auth layer put it there, and nothing
|
/// Handlers take `User` to say they need one; the auth layer put it there, and nothing
|
||||||
/// reaches a handler without passing through it.
|
/// reaches a handler without passing through it.
|
||||||
impl<S: Send + Sync> axum::extract::FromRequestParts<S> for crate::db::User {
|
impl<S: Send + Sync> axum::extract::FromRequestParts<S> for crate::db::User {
|
||||||
@@ -276,6 +272,7 @@ async fn login(
|
|||||||
let user = user.expect("verified above");
|
let user = user.expect("verified above");
|
||||||
let token = crate::auth::new_session_token();
|
let token = crate::auth::new_session_token();
|
||||||
state.ctx.db.create_session(user.id, &token)?;
|
state.ctx.db.create_session(user.id, &token)?;
|
||||||
|
state.ctx.db.signed_in(user.id)?;
|
||||||
tracing::info!(user = %user.name, "signed in");
|
tracing::info!(user = %user.name, "signed in");
|
||||||
|
|
||||||
let days = state.ctx.cfg().web.session_days.max(1);
|
let days = state.ctx.cfg().web.session_days.max(1);
|
||||||
@@ -306,8 +303,15 @@ async fn logout(State(state): State<WebState>, req: Request) -> Response {
|
|||||||
resp
|
resp
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn me(user: crate::db::User) -> Json<serde_json::Value> {
|
/// Who is signed in, and, for someone the proxy signed in, where Sign out should send them.
|
||||||
Json(serde_json::json!({ "name": user.name, "admin": user.is_admin }))
|
async fn me(
|
||||||
|
State(state): State<WebState>,
|
||||||
|
user: crate::db::User,
|
||||||
|
axum::Extension(Proxied(by_proxy)): axum::Extension<Proxied>,
|
||||||
|
) -> Json<serde_json::Value> {
|
||||||
|
let url = state.ctx.cfg().web.sign_out_url.clone();
|
||||||
|
let sign_out = (by_proxy && !url.is_empty()).then_some(url);
|
||||||
|
Json(serde_json::json!({ "name": user.name, "admin": user.is_admin, "sign_out": sign_out }))
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- accounts: admin only ----
|
// ---- accounts: admin only ----
|
||||||
@@ -340,6 +344,7 @@ async fn list_users(
|
|||||||
.map(|u| {
|
.map(|u| {
|
||||||
serde_json::json!({
|
serde_json::json!({
|
||||||
"id": u.id, "name": u.name, "admin": u.is_admin, "password": u.pass_hash.is_some(),
|
"id": u.id, "name": u.name, "admin": u.is_admin, "password": u.pass_hash.is_some(),
|
||||||
|
"created": u.created, "last_login": u.last_login,
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
@@ -429,8 +434,22 @@ async fn remove_user(
|
|||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn login_page() -> Html<&'static str> {
|
/// The password form, except for someone the proxy vouches for: they are signed in already, and
|
||||||
Html(include_str!("../web/login.html"))
|
/// the form only made it look as if they were not.
|
||||||
|
async fn login_page(State(state): State<WebState>, req: Request) -> Response {
|
||||||
|
if vouched_name(&state.ctx.cfg(), &req).is_some() {
|
||||||
|
return Redirect::to("/").into_response();
|
||||||
|
}
|
||||||
|
Html(include_str!("../web/login.html")).into_response()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The 2004 icon, served once for both pages rather than inlined as base64 into each. The
|
||||||
|
/// sign-in page shows it, so it sits outside the auth layer with /login.
|
||||||
|
async fn icon() -> impl IntoResponse {
|
||||||
|
(
|
||||||
|
[(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")],
|
||||||
|
include_bytes!("../web/ipodderx-icon.png").as_slice(),
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn constant_time_eq(a: &str, b: &str) -> bool {
|
fn constant_time_eq(a: &str, b: &str) -> bool {
|
||||||
@@ -495,6 +514,9 @@ async fn feeds(
|
|||||||
let mut out = Vec::with_capacity(mine.len());
|
let mut out = Vec::with_capacity(mine.len());
|
||||||
for sub in &subs {
|
for sub in &subs {
|
||||||
let (id, feed) = (&sub.id, &sub.cfg);
|
let (id, feed) = (&sub.id, &sub.cfg);
|
||||||
|
// In a group, what you have not set on the feed comes from your settings on the group,
|
||||||
|
// the same fallback the scanner uses (`Db::subscribers`).
|
||||||
|
let up = feed.group.as_deref().and_then(|g| mine.get(g));
|
||||||
let Some(mine) = mine.get(id) else { continue };
|
let Some(mine) = mine.get(id) else { continue };
|
||||||
let s = state.ctx.db.feed_summary(id)?;
|
let s = state.ctx.db.feed_summary(id)?;
|
||||||
let st = state.ctx.db.http_state(id)?;
|
let st = state.ctx.db.http_state(id)?;
|
||||||
@@ -504,11 +526,22 @@ async fn feeds(
|
|||||||
title: s.title,
|
title: s.title,
|
||||||
image: s.image,
|
image: s.image,
|
||||||
folder: feed.folder.clone(),
|
folder: feed.folder.clone(),
|
||||||
keywords: mine.keywords.clone().unwrap_or_else(|| feed.keywords.clone()),
|
keywords: mine
|
||||||
allow_explicit: mine.allow_explicit.unwrap_or(feed.allow_explicit),
|
.keywords
|
||||||
auto_download: mine.auto_download.unwrap_or(feed.auto_download),
|
.clone()
|
||||||
|
.or_else(|| up.and_then(|u| u.keywords.clone()))
|
||||||
|
.unwrap_or_else(|| feed.keywords.clone()),
|
||||||
|
allow_explicit: mine
|
||||||
|
.allow_explicit
|
||||||
|
.or(up.and_then(|u| u.allow_explicit))
|
||||||
|
.unwrap_or(feed.allow_explicit),
|
||||||
|
auto_download: mine
|
||||||
|
.auto_download
|
||||||
|
.or(up.and_then(|u| u.auto_download))
|
||||||
|
.unwrap_or(feed.auto_download),
|
||||||
max_new_per_check: mine
|
max_new_per_check: mine
|
||||||
.max_new_per_check
|
.max_new_per_check
|
||||||
|
.or(up.and_then(|u| u.max_new_per_check))
|
||||||
.map(|n| n as usize)
|
.map(|n| n as usize)
|
||||||
.or(feed.max_new_per_check),
|
.or(feed.max_new_per_check),
|
||||||
group: feed.group.clone(),
|
group: feed.group.clone(),
|
||||||
@@ -580,24 +613,33 @@ struct PopularRow {
|
|||||||
|
|
||||||
/// Every feed that may be listed, with everyone counted, you included, most subscribers
|
/// Every feed that may be listed, with everyone counted, you included, most subscribers
|
||||||
/// first. Popular is the top of it, the directory is all of it, and it is all that
|
/// first. Popular is the top of it, the directory is all of it, and it is all that
|
||||||
/// `subscribe_popular` will subscribe you to.
|
/// `subscribe_popular` will subscribe you to. An OPML or a Patreon creator is listed as the
|
||||||
|
/// feeds inside it and never itself: both lists are for finding a show.
|
||||||
fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
|
fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
|
||||||
let db = &state.ctx.db;
|
let db = &state.ctx.db;
|
||||||
let mine: std::collections::HashSet<String> =
|
let mine: std::collections::HashSet<String> =
|
||||||
db.subscriptions_for(user_id)?.into_iter().map(|s| s.feed_id).collect();
|
db.subscriptions_for(user_id)?.into_iter().map(|s| s.feed_id).collect();
|
||||||
let counts = db.subscriber_counts()?;
|
let counts = db.subscriber_counts()?;
|
||||||
|
let catalogue = crate::subscriptions(&state.ctx)?;
|
||||||
|
let by_id: std::collections::HashMap<&str, &crate::config::Feed> =
|
||||||
|
catalogue.iter().map(|s| (s.id.as_str(), &s.cfg)).collect();
|
||||||
|
let is_folder: std::collections::HashSet<&str> =
|
||||||
|
catalogue.iter().filter_map(|s| s.cfg.group.as_deref()).collect();
|
||||||
let mut out = vec![];
|
let mut out = vec![];
|
||||||
for s in crate::subscriptions(&state.ctx)? {
|
for s in &catalogue {
|
||||||
let n = counts.get(&s.id).copied().unwrap_or(0);
|
let n = counts.get(&s.id).copied().unwrap_or(0);
|
||||||
// A feed from an OPML rides on the OPML: everyone subscribed to it counts every feed
|
// A feed inside an OPML that looks private is as private as the OPML.
|
||||||
// inside, which would bury everything anyone chose on purpose.
|
let folder = s.cfg.group.as_deref().and_then(|g| by_id.get(g));
|
||||||
let from_opml = s.managed || s.cfg.group.is_some();
|
if n == 0
|
||||||
if n == 0 || from_opml || looks_private(&s.cfg) {
|
|| is_folder.contains(s.id.as_str())
|
||||||
|
|| looks_private(&s.cfg)
|
||||||
|
|| folder.is_some_and(|f| looks_private(f))
|
||||||
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let sum = db.feed_summary(&s.id)?;
|
let sum = db.feed_summary(&s.id)?;
|
||||||
let subscribed = mine.contains(&s.id);
|
let subscribed = mine.contains(&s.id);
|
||||||
out.push(PopularRow { id: s.id, title: sum.title, image: sum.image, subscribers: n, subscribed });
|
out.push(PopularRow { id: s.id.clone(), title: sum.title, image: sum.image, subscribers: n, subscribed });
|
||||||
}
|
}
|
||||||
out.sort_by(|a, b| b.subscribers.cmp(&a.subscribers).then_with(|| sort_name(a).cmp(&sort_name(b))));
|
out.sort_by(|a, b| b.subscribers.cmp(&a.subscribers).then_with(|| sort_name(a).cmp(&sort_name(b))));
|
||||||
Ok(out)
|
Ok(out)
|
||||||
@@ -612,7 +654,7 @@ async fn get_popular(
|
|||||||
Ok(Json(rows))
|
Ok(Json(rows))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Every feed that may be listed, A to Z.
|
/// Every feed that may be listed, A to Z, with the feeds inside an OPML in place of the OPML.
|
||||||
async fn get_directory(
|
async fn get_directory(
|
||||||
State(state): State<WebState>,
|
State(state): State<WebState>,
|
||||||
user: crate::db::User,
|
user: crate::db::User,
|
||||||
@@ -634,7 +676,7 @@ async fn subscribe_popular(
|
|||||||
Path(id): Path<String>,
|
Path(id): Path<String>,
|
||||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||||
if !popular(&state, user.id)?.iter().any(|p| p.id == id) {
|
if !popular(&state, user.id)?.iter().any(|p| p.id == id) {
|
||||||
return Err(ApiError::bad_request(format!("{id:?} is not on the popular list")));
|
return Err(ApiError::bad_request(format!("{id:?} is not in the directory")));
|
||||||
}
|
}
|
||||||
state.ctx.db.subscribe(user.id, &id)?;
|
state.ctx.db.subscribe(user.id, &id)?;
|
||||||
Ok(Json(serde_json::json!({ "id": id })))
|
Ok(Json(serde_json::json!({ "id": id })))
|
||||||
@@ -739,7 +781,14 @@ mod tests {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn only_the_last_admin_is_protected() {
|
fn only_the_last_admin_is_protected() {
|
||||||
let u = |id, is_admin| crate::db::User { id, name: format!("u{id}"), pass_hash: None, is_admin };
|
let u = |id, is_admin| crate::db::User {
|
||||||
|
id,
|
||||||
|
name: format!("u{id}"),
|
||||||
|
pass_hash: None,
|
||||||
|
is_admin,
|
||||||
|
created: None,
|
||||||
|
last_login: None,
|
||||||
|
};
|
||||||
assert!(last_admin(&[u(1, true), u(2, false)], 1));
|
assert!(last_admin(&[u(1, true), u(2, false)], 1));
|
||||||
assert!(!last_admin(&[u(1, true), u(2, true)], 1), "another admin remains");
|
assert!(!last_admin(&[u(1, true), u(2, true)], 1), "another admin remains");
|
||||||
assert!(!last_admin(&[u(1, true), u(2, false)], 2), "not an admin at all");
|
assert!(!last_admin(&[u(1, true), u(2, false)], 2), "not an admin at all");
|
||||||
@@ -803,15 +852,6 @@ mod tests {
|
|||||||
"another feed already has that URL"
|
"another feed already has that URL"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn generated_tokens_are_32_hex_chars_and_not_repeated() {
|
|
||||||
let a = generate_token();
|
|
||||||
let b = generate_token();
|
|
||||||
assert_eq!(a.len(), 32);
|
|
||||||
assert!(a.chars().all(|c| c.is_ascii_hexdigit()));
|
|
||||||
assert_ne!(a, b);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
@@ -893,6 +933,19 @@ struct NewFeed {
|
|||||||
folder: Option<String>,
|
folder: Option<String>,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
keywords: Vec<String>,
|
keywords: Vec<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
allow_explicit: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it
|
||||||
|
/// goes on your subscription, and before the first scan, which would otherwise skip every
|
||||||
|
/// explicit item.
|
||||||
|
fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
|
||||||
|
if allow {
|
||||||
|
let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() };
|
||||||
|
state.ctx.db.set_subscription(user_id, &sub)?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn add_feed(
|
async fn add_feed(
|
||||||
@@ -901,23 +954,28 @@ async fn add_feed(
|
|||||||
Json(body): Json<NewFeed>,
|
Json(body): Json<NewFeed>,
|
||||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||||
let mut cfg = (*state.ctx.cfg()).clone();
|
let mut cfg = (*state.ctx.cfg()).clone();
|
||||||
|
let url = crate::feed::expand_input(&body.url);
|
||||||
// Someone else may already have it. Then adding costs nothing: no second fetch, no
|
// Someone else may already have it. Then adding costs nothing: no second fetch, no
|
||||||
// second copy on disk, just another name against the same feed.
|
// second copy on disk, just another name against the same feed.
|
||||||
if let Some(existing) = crate::subscriptions(&state.ctx)?
|
if let Some(existing) = crate::subscriptions(&state.ctx)?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.find(|s| s.cfg.url == body.url)
|
.find(|s| crate::feed::same_feed(&s.cfg.url, &url))
|
||||||
{
|
{
|
||||||
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some();
|
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some();
|
||||||
state.ctx.db.subscribe(user.id, &existing.id)?;
|
state.ctx.db.subscribe(user.id, &existing.id)?;
|
||||||
|
if !already {
|
||||||
|
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit)?;
|
||||||
|
}
|
||||||
scan_soon(&state, Some(existing.id.clone())).await;
|
scan_soon(&state, Some(existing.id.clone())).await;
|
||||||
return Ok(Json(
|
return Ok(Json(
|
||||||
serde_json::json!({ "id": existing.id, "existing": already }),
|
serde_json::json!({ "id": existing.id, "existing": already }),
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
let id = crate::add_one(&state.ctx, &mut cfg, &body.url, body.folder, body.keywords).await?;
|
let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
|
||||||
cfg.save(&state.config_path)?;
|
cfg.save(&state.config_path)?;
|
||||||
state.ctx.reload_cfg(&state.config_path)?;
|
state.ctx.reload_cfg(&state.config_path)?;
|
||||||
state.ctx.db.subscribe(user.id, &id)?;
|
state.ctx.db.subscribe(user.id, &id)?;
|
||||||
|
explicit_on_add(&state, user.id, &id, body.allow_explicit)?;
|
||||||
scan_soon(&state, Some(id.clone())).await;
|
scan_soon(&state, Some(id.clone())).await;
|
||||||
Ok(Json(serde_json::json!({ "id": id, "existing": false })))
|
Ok(Json(serde_json::json!({ "id": id, "existing": false })))
|
||||||
}
|
}
|
||||||
@@ -1074,7 +1132,7 @@ async fn remove_feed(
|
|||||||
{
|
{
|
||||||
state.ctx.db.unsubscribe(user.id, &child.id)?;
|
state.ctx.db.unsubscribe(user.id, &child.id)?;
|
||||||
}
|
}
|
||||||
if state.ctx.db.subscriber_count(&id)? > 0 {
|
if state.ctx.db.subscriber_counts()?.contains_key(&id) {
|
||||||
return Ok(StatusCode::NO_CONTENT);
|
return Ok(StatusCode::NO_CONTENT);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1164,9 +1222,9 @@ async fn delete_file(
|
|||||||
let complaint = match (starred, unread) {
|
let complaint = match (starred, unread) {
|
||||||
(0, 0) => None,
|
(0, 0) => None,
|
||||||
(0, u) => Some(format!("{} subscribed to this feed {} not played it yet", people(u), if u == 1 { "has" } else { "have" })),
|
(0, u) => Some(format!("{} subscribed to this feed {} not played it yet", people(u), if u == 1 { "has" } else { "have" })),
|
||||||
(st, 0) => Some(format!("another {} starred it to keep", people(st))),
|
(st, 0) => Some(format!("another {} kept it", people(st))),
|
||||||
(st, u) => Some(format!(
|
(st, u) => Some(format!(
|
||||||
"another {} starred it to keep, and {} not played it yet",
|
"another {} kept it, and {} not played it yet",
|
||||||
people(st),
|
people(st),
|
||||||
if u == 1 { "one person has".to_string() } else { format!("{u} have") }
|
if u == 1 { "one person has".to_string() } else { format!("{u} have") }
|
||||||
)),
|
)),
|
||||||
@@ -1215,9 +1273,20 @@ async fn fetch_now(
|
|||||||
|
|
||||||
/// The same broadcast the socket clients read, as server-sent events.
|
/// The same broadcast the socket clients read, as server-sent events.
|
||||||
async fn events(State(state): State<WebState>) -> Sse<impl futures_util::Stream<Item = Result<SseEvent, std::convert::Infallible>>> {
|
async fn events(State(state): State<WebState>) -> Sse<impl futures_util::Stream<Item = Result<SseEvent, std::convert::Infallible>>> {
|
||||||
let stream = BroadcastStream::new(state.events.subscribe()).filter_map(|ev| async move {
|
// A client that falls behind skips what it missed rather than being cut off.
|
||||||
let ev = ev.ok()?;
|
let stream = futures_util::stream::unfold(state.events.subscribe(), |mut rx| async move {
|
||||||
Some(Ok(SseEvent::default().data(serde_json::to_string(&ev).ok()?)))
|
loop {
|
||||||
|
match rx.recv().await {
|
||||||
|
Ok(ev) => {
|
||||||
|
if let Ok(data) = serde_json::to_string(&ev) {
|
||||||
|
let ev = Ok::<_, std::convert::Infallible>(SseEvent::default().data(data));
|
||||||
|
return Some((ev, rx));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(broadcast::error::RecvError::Lagged(_)) => {}
|
||||||
|
Err(broadcast::error::RecvError::Closed) => return None,
|
||||||
|
}
|
||||||
|
}
|
||||||
});
|
});
|
||||||
Sse::new(stream).keep_alive(axum::response::sse::KeepAlive::default())
|
Sse::new(stream).keep_alive(axum::response::sse::KeepAlive::default())
|
||||||
}
|
}
|
||||||
@@ -1428,8 +1497,6 @@ async fn patch_settings(
|
|||||||
)));
|
)));
|
||||||
}
|
}
|
||||||
cfg.general.schedule = sched;
|
cfg.general.schedule = sched;
|
||||||
// The legacy key would otherwise keep shadowing intent in the file.
|
|
||||||
cfg.general.interval_mins = None;
|
|
||||||
}
|
}
|
||||||
if let Some(v) = body.max_new_per_check {
|
if let Some(v) = body.max_new_per_check {
|
||||||
cfg.general.max_new_per_check = v;
|
cfg.general.max_new_per_check = v;
|
||||||
|
|||||||
@@ -142,7 +142,7 @@ test('the filter tabs change what is listed', async ({ page }) => {
|
|||||||
await page.locator('.tabs button', { hasText: 'Unread' }).first().click();
|
await page.locator('.tabs button', { hasText: 'Unread' }).first().click();
|
||||||
expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all);
|
expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all);
|
||||||
|
|
||||||
await page.locator('.tabs button', { hasText: 'Flagged' }).first().click();
|
await page.locator('.tabs button', { hasText: 'Kept' }).first().click();
|
||||||
await expect(page.locator('#count')).toContainText('0 items');
|
await expect(page.locator('#count')).toContainText('0 items');
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -187,9 +187,11 @@ test('an OPML subscription is a collapsible folder', async ({ page }) => {
|
|||||||
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
|
||||||
}));
|
}));
|
||||||
|
|
||||||
// Every row reserves the chevron slot for alignment; only a folder's is clickable.
|
// Only a folder has a triangle, and it is a button that says whether the folder is open.
|
||||||
const chev = page.locator('.feed.group .chev');
|
const chev = page.locator('.feed.group .chev');
|
||||||
await expect(chev).toBeVisible({ timeout: 20_000 });
|
await expect(chev).toBeVisible({ timeout: 20_000 });
|
||||||
|
await expect(page.locator('.feed:not(.group) .chev')).toHaveCount(0);
|
||||||
|
await expect(chev).toHaveAttribute('aria-expanded', 'false');
|
||||||
|
|
||||||
// Closed by default: the children are not listed until the folder is opened.
|
// Closed by default: the children are not listed until the folder is opened.
|
||||||
const before = await page.locator('.feed').count();
|
const before = await page.locator('.feed').count();
|
||||||
@@ -346,6 +348,10 @@ test('a second person has their own feeds and their own read state', async ({ br
|
|||||||
const ctx = await browser.newContext();
|
const ctx = await browser.newContext();
|
||||||
const page = await ctx.newPage();
|
const page = await ctx.newPage();
|
||||||
await page.goto('/login');
|
await page.goto('/login');
|
||||||
|
// The sign-in page shows the icon, so it has to load before anyone has signed in.
|
||||||
|
const icon = await page.request.get('/icon.png');
|
||||||
|
expect(icon.status()).toBe(200);
|
||||||
|
expect(icon.headers()['content-type']).toBe('image/png');
|
||||||
await page.locator('#name').fill('sam');
|
await page.locator('#name').fill('sam');
|
||||||
await page.locator('#pw').fill('sampassword');
|
await page.locator('#pw').fill('sampassword');
|
||||||
await page.locator('button[type=submit]').click();
|
await page.locator('button[type=submit]').click();
|
||||||
@@ -427,6 +433,9 @@ test('an admin adds someone, makes them an admin, and removes them', async ({ pa
|
|||||||
await page.locator('#uadd').click();
|
await page.locator('#uadd').click();
|
||||||
const row = userRow(page, 'pat');
|
const row = userRow(page, 'pat');
|
||||||
await expect(row).toBeVisible();
|
await expect(row).toBeVisible();
|
||||||
|
// When each account was added and last signed in; the admin signed in with the token link.
|
||||||
|
await expect(row).toContainText(/Added .* never signed in/);
|
||||||
|
await expect(userRow(page, 'admin')).toContainText(/signed in \d+m ago/);
|
||||||
await expect(row.locator('[data-a="admin"]')).not.toBeChecked();
|
await expect(row.locator('[data-a="admin"]')).not.toBeChecked();
|
||||||
|
|
||||||
await row.locator('[data-a="admin"]').check();
|
await row.locator('[data-a="admin"]').check();
|
||||||
@@ -607,8 +616,8 @@ test('Popular lists what everyone here reads, but never a private feed', async (
|
|||||||
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
|
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
|
||||||
const offered = piper.locator('#popular .childrow');
|
const offered = piper.locator('#popular .childrow');
|
||||||
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
|
||||||
// An OPML's own feeds ride on the OPML, and a key in a URL marks someone's paid feed.
|
// An OPML is listed as the feeds inside it, and a key in a URL marks someone's paid feed.
|
||||||
await expect(offered.filter({ hasText: /Grouped Show|grouped-show/ })).toHaveCount(0);
|
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
|
||||||
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
|
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
|
||||||
|
|
||||||
// No URL reaches the page at all, so neither can a key, and the server holds the same line.
|
// No URL reaches the page at all, so neither can a key, and the server holds the same line.
|
||||||
@@ -617,19 +626,25 @@ test('Popular lists what everyone here reads, but never a private feed', async (
|
|||||||
expect(listed).not.toContain('.xml');
|
expect(listed).not.toContain('.xml');
|
||||||
expect((await piper.request.post('/api/popular/paid-show')).status()).toBe(400);
|
expect((await piper.request.post('/api/popular/paid-show')).status()).toBe(400);
|
||||||
|
|
||||||
// Popular is the top ten of the directory, and the directory is every listed feed, A to Z.
|
// Popular is the top ten of the directory, and the directory is every listed feed A to Z,
|
||||||
|
// with an OPML's feeds in place of the OPML in both.
|
||||||
const dir = await (await piper.request.get('/api/directory')).json();
|
const dir = await (await piper.request.get('/api/directory')).json();
|
||||||
const top = await (await piper.request.get('/api/popular')).json();
|
const top = await (await piper.request.get('/api/popular')).json();
|
||||||
const names = dir.map(p => (p.title || p.id).toLowerCase());
|
const names = dir.map(p => (p.title || p.id).toLowerCase());
|
||||||
expect(names).toEqual([...names].sort());
|
expect(names).toEqual([...names].sort());
|
||||||
|
const ids = dir.map(p => p.id);
|
||||||
|
expect(ids).not.toContain('test-subscriptions');
|
||||||
|
expect(ids).toEqual(expect.arrayContaining(['grouped-show', 'aardvark-radio']));
|
||||||
expect(top.length).toBe(Math.min(10, dir.length));
|
expect(top.length).toBe(Math.min(10, dir.length));
|
||||||
expect(top.every(t => dir.some(d => d.id === t.id))).toBe(true);
|
expect(top.every(t => ids.includes(t.id))).toBe(true);
|
||||||
expect(dir.map(p => p.id)).not.toContain('paid-show');
|
expect(ids).not.toContain('paid-show');
|
||||||
|
|
||||||
// Subscribe from the directory this time; the popular list shares the same rows.
|
// Subscribe from the directory this time; the popular list shares the same rows.
|
||||||
await piper.locator('#feedlist .place', { hasText: 'Directory' }).click();
|
await piper.locator('#feedlist .place', { hasText: 'Directory' }).click();
|
||||||
await expect(piper.locator('#count')).toContainText(`Directory: ${dir.length} feed`);
|
await expect(piper.locator('#count')).toContainText(`Directory: ${dir.length} feed`);
|
||||||
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible();
|
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible();
|
||||||
|
await expect(offered.filter({ hasText: /Grouped Show|grouped-show/ })).toBeVisible();
|
||||||
|
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
|
||||||
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
|
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
|
||||||
|
|
||||||
const row = async () =>
|
const row = async () =>
|
||||||
@@ -778,3 +793,20 @@ test('play in the Files pane plays once, in the player bar', async ({ page }) =>
|
|||||||
await expect(page.locator('audio')).toHaveCount(1); // the player bar's, and nothing else
|
await expect(page.locator('audio')).toHaveCount(1); // the player bar's, and nothing else
|
||||||
await page.locator('#pclose').click();
|
await page.locator('#pclose').click();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('someone the proxy signs in never sees the password page, and signs out through the proxy', async ({ page, browser }) => {
|
||||||
|
// Signed in with the token, not by the proxy: Sign out stays ipx's own.
|
||||||
|
expect((await (await page.request.get('/api/me')).json()).sign_out).toBeNull();
|
||||||
|
|
||||||
|
const ctx = await browser.newContext({ extraHTTPHeaders: { 'X-Test-User': 'proxied@example.com' } });
|
||||||
|
const proxied = await ctx.newPage();
|
||||||
|
// Regression: after Sign out, the password form showed to someone the proxy still vouched for.
|
||||||
|
await proxied.goto('/login');
|
||||||
|
await expect(proxied).toHaveURL(/:8791\/$/);
|
||||||
|
await expect(proxied.locator('#who')).toContainText('proxied@example.com');
|
||||||
|
expect(await (await proxied.request.get('/api/me')).json())
|
||||||
|
.toMatchObject({ name: 'proxied@example.com', sign_out: '/signed-out-by-the-proxy' });
|
||||||
|
await proxied.locator('#signout').click();
|
||||||
|
await expect(proxied).toHaveURL(/\/signed-out-by-the-proxy$/);
|
||||||
|
await ctx.close();
|
||||||
|
});
|
||||||
|
|||||||
@@ -37,6 +37,10 @@ enabled = false
|
|||||||
enabled = true
|
enabled = true
|
||||||
bind = "127.0.0.1:8791"
|
bind = "127.0.0.1:8791"
|
||||||
token = "${TOKEN}"
|
token = "${TOKEN}"
|
||||||
|
# The proxy path, for tests that send the header themselves: the daemon sees them at 127.0.0.1.
|
||||||
|
trusted_header = "X-Test-User"
|
||||||
|
trusted_proxies = ["127.0.0.1"]
|
||||||
|
sign_out_url = "/signed-out-by-the-proxy"
|
||||||
|
|
||||||
[feeds.test-show]
|
[feeds.test-show]
|
||||||
url = "http://127.0.0.1:8792/show.xml"
|
url = "http://127.0.0.1:8792/show.xml"
|
||||||
|
|||||||
283
web/index.html
283
web/index.html
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user