26 Commits

Author SHA1 Message Date
be3820bbbd Release 0.5.3: OPML orphan scan, feed error UI, small UI fixes
- Stop scanning an OPML/Patreon feed's derived rows once nobody subscribes
  to it; retire them (drop or orphan) the way sync_group already does when
  the list itself drops one. This is what let 922 defunct davewiner feeds
  keep scanning hourly after the OPML left config.
- Repair feed XML with a bare `&`, and give a plain reason (moved web page
  with its new address when linked, or nothing yet for an empty body)
  instead of a raw parser error.
- Show a failing feed's plain-English reason and next step (Unsubscribe /
  Use the new address) in the sidebar and on its own page, once it has
  been down a day.
- Fix four small UI bugs: show-note links open in a new tab, video files
  play as video, an opened item no longer disappears from the Unread tab,
  and Subscribe/Unsubscribe get their own icons.
- Fix Settings disappearing for non-admin accounts: it was hiding the
  whole modal instead of just the admin-only parts (Users, the editable
  schedule/quota, Save), which are the only parts the server actually
  refuses them.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DmQfE1eFPApnXWyPHBWqUA
2026-09-14 14:53:45 +00:00
51ce0bf9eb TODO.md: show a publisher's error in the UI
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dcx59boh4pasuNwAVU7un
2026-09-13 13:13:15 +00:00
6ec900e456 TODO.md: the errors in the production log
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dcx59boh4pasuNwAVU7un
2026-09-13 13:06:16 +00:00
d4304869b6 TODO.md: cleared, the database trim and Popular are done
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 14:30:10 +00:00
9aae3097e7 Release 0.5.2
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 14:27:42 +00:00
2ff2074755 Answer status to the client that asked, not everyone
status is a terminal event. Broadcast, the healthcheck's answer ended any
ipx fetch that was watching a scan, which stopped reading at the next probe
while the scan carried on. It could not happen while status waited behind
the scan; answering it at once made it happen every 30 seconds. Each
connection's writer now takes private replies beside the broadcast, and
the test checks another client hears nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 14:27:42 +00:00
1698cf8d1e Answer status on the socket instead of queuing it behind the worker
The worker runs one job at a time, and status was one of its jobs, so the
Docker healthcheck waited behind the startup scan (54 seconds of it after
the last deploy) and timed out at 5. Any scan or download longer than
three probes would have had a working daemon marked unhealthy. The socket
now answers status straight away; everything else still queues. A test
fills the queue and checks status comes back anyway.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 14:19:00 +00:00
b94a74ef15 Sign out through the proxy when the proxy signed you in
Sign out cleared ipx's cookies and showed its password page, while
Cloudflare Access still vouched for the person: nothing was signed out,
and the page looked like the wrong login. /api/me now says, for someone
the proxy signed in, where to go instead ([web] sign_out_url, which is
/cdn-cgi/access/logout behind Access), and /login sends anyone the proxy
vouches for on to their feeds. The header check both use is one function.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 14:10:30 +00:00
9a8a3c696f docs: the ipodderx tile in Authentik's library
A bookmark application with no provider, so ipodderx shows in the library
beside Outline. Recorded with its id and how to delete it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 14:03:06 +00:00
bedf64e645 docs/sso.md: the sign-in setup ipodderx.sdf1.net really runs
Authentik is Cloudflare Access's OpenID Connect identity provider, not
something in the request path, and the tunnel's requests reach ipx from
the content_default gateway, 192.168.16.1, not 127.0.0.1. The page is
rewritten from what was measured, with checks for both the trusted and
the refused path, and docs/history.md records every change made to get
there with how to undo it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 13:58:18 +00:00
586d2c07a1 ipx user rename: give an account the name the proxy signs it in as
An account made by hand before the proxy was set up is called what it was
given ('rays'), while Cloudflare Access vouches for an email address. With
auto_create_users on, the first visit through the tunnel would make a
second, empty account. Renaming keeps the id, so feeds, read state and
admin rights go with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 13:54:29 +00:00
2ba83c3aed Keep when each account was added and when it last signed in
users.created comes back, beside a new last_login, for whoever maintains
the server. A password sign-in, the token link and a request through the
proxy all count, recorded to the hour so the proxy's per-request vouching
is not a write each time. Settings -> Users and ipx user list show both.
The three user queries now share one row mapping.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 13:37:09 +00:00
1352f0d54d Show notes cut off mid-tag give way to the item's description
libsyn served Daily Meditation Podcast's content:encoded cut at the '>'
inside a Tailwind class pasted from a web app, so 57 items began halfway
through a tag and the page showed the rest of it as text. Their
description was whole. A body that closes an attribute list before any
tag opens now falls back to the description, for RSS and Atom alike.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 13:29:00 +00:00
a958f7cb37 Trim the state database; Popular lists feeds the way Directory does
Drops the created columns on users, subscriptions and sessions, which were
written by every insert and read by nothing, and migrate()'s add list, whose
columns all predate 0.3.0. Removes Db::subscribed_feed_ids (no callers),
Db::subscriber_count (one caller wanting > 0) and Managed.orphaned (never
read). The old-database test now builds the tables with foreign keys on.

Popular now lists the feeds inside an OPML or a Patreon creator, never the
collection, as Directory does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 13:15:55 +00:00
457a58dcc5 Directory lists the feeds inside an OPML, not the OPML
Popular still counts an OPML as one feed, since everyone subscribed to it
counts for every feed inside and they would bury the rest. The directory is
for finding a show, so it lists them one by one and never the OPML. A feed
inside an OPML that looks private is hidden with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 02:26:52 +00:00
2af57065c6 Release 0.5.1
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 02:15:53 +00:00
564b011c7a Give the folder triangle a wider gutter
The feed list's left padding grows from 8 to 16 px and the triangle's
button from 16 to 24 px wide, so it is no longer cramped against the
folder's art. Everything in the list shifts together, so feeds still line
up with the places above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 02:13:32 +00:00
9eb7aadced Release 0.5.0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 02:06:23 +00:00
c6bceaef37 Docs: a slow migration and a CLI run at the same time
Every ipx command migrates when it opens the database, so the healthcheck
collided with the daemon while it dropped the old entries columns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 01:59:05 +00:00
dc63d6acaf Cut what the audit found: dead columns, one-time upgrades, three deps
Works through TODO.md from the 2026-09-12 over-engineering audit. Drops the
entries.read/flagged/position columns (migrate() removes them from older
databases), migrate_opml_children, the legacy interval_mins key, the
contrib/ systemd units, test-only Db wrappers, a duplicate token generator,
redundant logbuf visitors, unused page state and CSS, and the infer, dirs
and tokio-stream dependencies. The icon is served once as /icon.png instead
of inlined four times, taking about 94 KB off the two pages.

The adoption's subscription half was not dead: it gives a fresh install's
first admin the config's feeds. It stays as adopt_catalogue, now tested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 01:55:44 +00:00
8937f35f00 TODO.md: cleared, the design pass is done
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-12 01:31:08 +00:00
0990f2a90d Design pass on the web UI: amber for new, EQ bars, keyboard feed list
Works through TODO.md from the 2026-09-11 review. Unread badges, dots and
download bars take the icon's amber; the playing item is marked by EQ bars
that move only while it plays. The feed list is usable from the keyboard,
focus rings show everywhere, and folders get a mosaic of their shows' art
with the triangle hung in the margin. Sentence-case labels, fewer bold
weights, tinted initials tiles, shorter header lines, "Kept" everywhere,
and the list gets the room the empty panes had. Reduced motion is honoured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAC7sLVqfKmY6rsTLXzNgk
2026-09-11 22:24:41 +00:00
0cc002cdfa todo.md: folders in the sidebar; drop the system-theme item
The disclosure triangle, looked at closely: the feed list cannot be
reached from the keyboard (span and div, no tabindex), every feed is
pushed 28 px right for a slot only folders use, the target is 18 px,
shows barely nest under their folder, a folder looks like a feed, and
a selected feed's placeholder tile vanishes in Dark and Light.

Following the system light or dark setting is off the list, by
choice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wi22VSVrkAvqNj61eqHsm9
2026-09-11 19:39:49 +00:00
2c9e899762 todo.md: the design review's list, to do later
A review of the web UI against screenshots of every view in all three
themes. The palette and Classic carry the iPodderX identity; Dark and
Light do not. First three: amber for new (unread badges, dots,
download progress), EQ bars as the playing marker, and toolbar focus
rings that overflow:hidden currently clips. The rest is weights,
sentence-case labels, pane sizes, header lines, wording, system theme,
reduced motion, art, sign-in and the Settings export/import icons.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wi22VSVrkAvqNj61eqHsm9
2026-09-11 18:53:42 +00:00
2e416f96cf Patreon creators split into their shows; filters follow settings
A Patreon token pasted into Add feed, or a creator link without
&show=, becomes a folder of that creator's shows, found through
Patreon's web API and kept in step like a subscribed OPML (sync_group,
split out of sync_opml). A creator already read as one feed is split
too: each show takes over the files and read state it held
(Db::adopt). A creator with one show stays a plain feed.

Filter verdicts are judged again every scan, so turning on Allow
explicit brings skipped items back. Add feed has an explicit box.
Feeds in a group follow your settings on the group, as its dialog
said. A new feed no longer takes the id of a removed one at a
different URL and shows its old items. See CHANGELOG.md [Unreleased]
and docs/history.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wi22VSVrkAvqNj61eqHsm9
2026-09-11 18:24:47 +00:00
9269aa99f7 README: a short overview that points into docs/
It described the layout from before 0.4.0 (items across the top, a
player below), and carried long sections on OPML, the log view and
this server's own deploy steps, all of which docs/ and CLAUDE.md cover.
It now says what ipx does, how to run it with Docker or from source,
the first sign-in, the TLS caveat, where the docs are, and the tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0173mGu6rK18Ne7UGTwAaVJV
2026-09-11 17:36:34 +00:00
28 changed files with 2240 additions and 869 deletions

View File

@@ -10,6 +10,139 @@ The long form, with what was wrong before and how it was found, is in
## [Unreleased]
## [0.5.3] - 2026-09-14
### Added
- A feed that has been failing for a day shows a plain-English reason in the sidebar and on its
own page, sorted from a 404, a 401/403, a 402, a name that no longer resolves, or a web page in
place of the feed -- with Unsubscribe or, when the page links its new feed, Use the new address.
A feed that fails once and reads fine again within a day is never flagged.
### Changed
- Unsubscribing from the last person's OPML or Patreon subscription now retires the feeds it
listed, the same as a feed the list itself drops: removed if nothing was downloaded, kept and
marked orphaned otherwise. Until now they stayed in the database and kept being scanned hourly
with auto-download on, which is how 922 defunct `davewiner` feeds outlived the OPML that listed
them.
### Fixed
- A feed whose XML uses a bare `&` instead of `&amp;` (kcpw, both feedland feeds) is now read
instead of refused.
- A feed URL that now serves a web page says so, and names the feed the page links to when it has
one, instead of a raw XML parser error.
- A publisher answering with an empty body (British Antarctic Survey's 202) is read as nothing new
to report, not a parse failure.
- A link in an item's show notes opens in a new tab instead of navigating away from ipx.
- A video file plays as video, in a small floating pane above the player bar, instead of silently
as sound only.
- On the Unread tab, opening an item no longer makes it disappear from the list -- it stays until
you open a different one, even if a scan finishes and refreshes the list while it is open.
- Subscribe and Unsubscribe have their own icons (a circled check and a circled minus) instead of
sharing the generic plus and minus used for adding feeds, users and imports.
- Settings no longer disappears for a non-admin account. It was hiding the whole Settings modal
along with the log and the users screen, but a non-admin has settings of their own in there --
their subscriptions' Export and Import, and the schedule and quota are worth seeing even without
a say in them. Only the log and the users screen, which the server also refuses them, are gone.
## [0.5.2] - 2026-09-12
### Added
- Settings → Users and `ipx user list` show when each account was added and when it last signed
in, to the hour.
- `ipx user rename <name> <new name>` renames an account and keeps its feeds, read state and admin
rights. An account made before the proxy was set up can take the name the proxy signs it in as.
### Changed
- Directory and Popular list the feeds inside an OPML one by one, and no longer the OPML itself,
so you can subscribe to just the shows you want.
- The database no longer records when subscriptions and sign-in sessions were created. Nothing
ever read it, and an existing database drops the columns on its next start.
### Fixed
- Show notes that the podcast's host cut off in the middle of a tag no longer open with a scrap of
HTML: the item's other copy of its notes is used instead, from the next time the feed changes.
Daily Meditation Podcast had 57.
- Docker no longer shows ipodderx as starting, or calls it unhealthy, while it scans or downloads:
`ipx status` answers at once instead of waiting for the job in progress to finish.
- Signing out after signing in through Cloudflare Access no longer lands on ipodderx's own password
page. With the new `sign_out_url` set, Sign out ends the Access session, and the password page
sends anyone the proxy signs in straight to their feeds.
- The sign-in guide, `docs/sso.md`, describes the setup ipodderx.sdf1.net really runs: Authentik as
Cloudflare Access's identity provider, and how to find the address ipx has to trust. It had never
been checked against a real setup, and pointed at the wrong address.
## [0.5.1] - 2026-09-12
### Fixed
- The triangle that opens an OPML or Patreon folder was cramped against the folder's art. It has
more room now, and a wider target to click.
## [0.5.0] - 2026-09-12
### Added
- A Patreon token pasted into Add feed, or a creator's RSS link without `&show=`, becomes a folder
of that creator's shows, kept in step on every scan like a subscribed OPML. A creator with only
one show stays a plain feed. One already added as a single long feed is split into its shows on
its next scan, keeping its files and what you had read.
- Add feed has an "Allow items marked explicit" box, so a new feed's first scan no longer skips
every explicit item.
### Changed
- Unread counts, unread dots and download progress are amber, the colour of the icon's EQ bars.
Blue is kept for the primary action and links, so a count no longer looks like a button.
- What is playing is marked by small EQ bars, in its row and in the player. They move only while it
plays.
- A folder in the sidebar shows its first four shows' art as a mosaic, and its shows sit under its
title. Only folders have a triangle, so every feed lines up with Directory and Popular above.
- Feeds without art get initials in a colour of their own, instead of all the same grey.
- The Flagged tab is called Kept, as the Keep button and Settings already said.
- A feed's header is one short line; when it checks next is in its tooltip.
- The item list takes more of the window, and the Files pane shows only when the item has files.
- Column headings and tags are in sentence case, and fewer things are bold.
- Unsubscribe is a round button beside the feed's other actions.
- Export and Import in Settings say what they do.
- The sign-in page shows the original icon large.
- Nothing animates when your system asks for reduced motion.
- The pages are about 90 KB smaller: the icon is served once instead of written into each.
- A web token generated for a new install is 64 characters instead of 32.
- The README is a short overview of what ipx does and how to run it, and points into `docs/` for
the rest. It still described the layout from before 0.4.0.
### Removed
- The systemd units in `contrib/`. Run ipx with Docker, or point a unit of your own at
`ipx daemon`.
- Upgrading from before 0.3.0 directly: what was read, kept or part-played before accounts is no
longer carried over to the admin, and OPML feeds that old versions wrote into `config.toml` are
no longer moved out of it. Upgrade through 0.4.0 first.
- `interval_mins` in `config.toml` is ignored; use `schedule`.
### Fixed
- The feed list works from the keyboard: Tab reaches every feed, Enter opens it, and Right and Left
open and close a folder. Every button shows where the focus is, including in the toolbar, which
used to clip the ring.
- "1 items" reads "1 item".
- A selected feed without art no longer loses its initials tile in Dark and Light.
- The player shows the feed's initials when there is no art, not the episode's.
- Turning on Allow explicit, or changing keywords or auto-download, brings back what those settings
had skipped on the feed's next scan. Before, an item was judged once, when first seen, and a
skipped one stayed skipped whatever you changed.
- Feeds inside an OPML or a Patreon creator follow your settings on the folder unless you set their
own, as the folder's settings dialog said they did. Before, the folder's settings reached nothing
inside it.
- A new feed no longer takes the name of one you removed earlier and shows that feed's old items.
Re-adding the same feed still gets its old name, and its history, back.
## [0.4.0] - 2026-09-11
### Added
@@ -199,7 +332,11 @@ The long form, with what was wrong before and how it was found, is in
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...main
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.3...main
[0.5.3]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.2...v0.5.3
[0.5.2]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.1...v0.5.2
[0.5.1]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.0...v0.5.1
[0.5.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.4.0...v0.5.0
[0.4.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.3.0...v0.4.0
[0.3.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.2.0...v0.3.0
[0.2.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.1.0...v0.2.0

View File

@@ -17,6 +17,7 @@ Arcane project `content`: `/mnt/fast/arcane/projects/content/compose.yaml`. That
| Database | `/mnt/user/ipodderx/state.db` | `/data/state.db` |
| Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` |
| Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` |
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | trusts `Cf-Access-Authenticated-User-Email` from `192.168.16.1`, the `content_default` gateway |
Deploying a change is: build and push the image, then pull it and recreate the container.
@@ -45,7 +46,10 @@ docker tag mirror.gcr.io/library/rust:1-slim-bookworm rust:1-slim-bookworm
Run those again now and then, or the local copies go stale.
The healthcheck runs `ipx status` against the control socket, so `(healthy)` in `docker ps` means
the worker is alive, not just the web port. The container restarts on its own after a reboot.
the daemon answers there and can read its database, not just that the web port is up. The socket
answers `status` itself instead of queuing it behind the worker's current job, so a long scan or
download does not fail the check; it also means a worker stuck on one job would still pass. The
container restarts on its own after a reboot.
Before the container, ipx ran by hand in code-server, with its files in `/config/.config/ipx/` and
`/config/.local/share/ipx/`. Those are still there and the container does not read them. If you run
@@ -99,9 +103,9 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
* **`entries.read`, `entries.flagged` and `entries.position` are dead columns.** Read state lives in
`entry_state` per user. Two bugs have already come from queries still reading the old ones
(retention, and the entry pruner) — grep before adding a third.
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
and `position` columns from before accounts; two bugs came from queries still reading them
(retention, and the entry pruner), and `migrate()` now drops them.
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
derived and never written to config.
@@ -113,8 +117,13 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
watch the shutdown channel itself; the daemon ignored SIGTERM for exactly this reason.
* Only one daemon per socket. Removing the socket file defeats the guard and you get two daemons
fighting over the database, with the stale one still holding the port.
* `/api/settings` answering `200` does **not** mean the worker is alive — it is a different task.
Probe the control socket (`ipx status`) to check that.
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
different task. `ipx status` checks the control socket and the database; to see the worker
getting through its jobs, watch for `scan complete` in the log.
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait
for `daemon started` in the log. Copy `state.db` aside before deploying one.
## House style

34
Cargo.lock generated
View File

@@ -436,17 +436,6 @@ dependencies = [
"shlex",
]
[[package]]
name = "cfb"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a347dcabdae9c31b0825fd6a8bed285ec9c2acb89c47827126d52fa4f59cece3"
dependencies = [
"fnv",
"uuid",
"web-time",
]
[[package]]
name = "cfg-if"
version = "1.0.4"
@@ -867,15 +856,6 @@ dependencies = [
"dirs-sys",
]
[[package]]
name = "dirs"
version = "7.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8d57d423b3c82e89b9a24ca3091fee61f456a26edbd28d26c65906f4bc1dcd8f"
dependencies = [
"dirs-sys",
]
[[package]]
name = "dirs-sys"
version = "0.5.0"
@@ -1608,15 +1588,6 @@ dependencies = [
"serde_core",
]
[[package]]
name = "infer"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f4200d433cbd5178df7797c9c2e75b348b728e39631cf14520d1e2fc424201f4"
dependencies = [
"cfb",
]
[[package]]
name = "intervaltree"
version = "0.2.7"
@@ -1634,7 +1605,7 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]]
name = "ipx"
version = "0.4.0"
version = "0.5.3"
dependencies = [
"ammonia",
"anyhow",
@@ -1643,9 +1614,7 @@ dependencies = [
"axum",
"chrono",
"clap",
"dirs",
"futures-util",
"infer",
"librqbit",
"opml",
"percent-encoding",
@@ -1656,7 +1625,6 @@ dependencies = [
"serde",
"serde_json",
"tokio",
"tokio-stream",
"toml",
"tower",
"tower-http 0.7.1",

View File

@@ -1,6 +1,6 @@
[package]
name = "ipx"
version = "0.4.0"
version = "0.5.3"
edition = "2024"
[dependencies]
@@ -11,9 +11,7 @@ atom_syndication = "0.12.10"
axum = "0.8.9"
chrono = { version = "0.4.45", default-features = false, features = ["std", "clock"] }
clap = { version = "4.6.6", features = ["derive"] }
dirs = "7.0.0"
futures-util = { version = "0.3.34", default-features = false, features = ["std"] }
infer = "0.22.0"
librqbit = { version = "9.0.1", default-features = false, features = ["rust-tls", "http-api-client"] }
opml = "1.1.6"
percent-encoding = "2.3.2"
@@ -24,7 +22,6 @@ rusqlite = { version = "0.40.2", features = ["bundled"] }
serde = { version = "1.0.229", features = ["derive"] }
serde_json = "1.0.151"
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
tokio-stream = { version = "0.1.19", features = ["sync"] }
toml = "1.1.5"
tower = { version = "0.5.3", features = ["util"] }
tower-http = { version = "0.7.1", features = ["fs"] }

144
README.md
View File

@@ -1,123 +1,83 @@
# ipodderx-rs
A headless podcatcher: scans RSS/Atom feeds, downloads enclosures (HTTP and BitTorrent), files them
into per-feed folders, and reaps old files to stay under a disk quota. Runs as a one-shot CLI or as
a daemon with a web UI, serving any number of people from one copy of the data.
A self-hosted podcatcher for a household. It checks your feeds, downloads the episodes, and serves
a web UI modelled on the 2004 Mac app **iPodderX**, for any number of people sharing one copy of
the files. One Rust binary, `ipx`, is both the daemon and the command line.
## Lineage
It is a rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python engine
behind iPodderX (2004-2008, Ray Slakinski & August Trometer).
A modern Rust rewrite of [ipodderx-core](https://git.sdf1.net/rays/ipodderx-core), the Python 2
engine behind **iPodderX** (2004-2008, Ray Slakinski & August Trometer), open-sourced under the MIT
License in 2010.
## What it does
What carries over: the feed scan and TTL handling, GUID/URL dedupe, per-feed and per-date download
folders, keyword filters, the explicit-content filter, torrent enclosures, and "SmartSpace" -- the
oldest-first disk quota reaper.
- **The web UI.** It has a toolbar, and a feed list that opens with Directory, Popular and All
Subscriptions. Items sit in a sortable table with a Files pane, and there is a player bar. It
comes in Dark, Light and Classic themes, and works on a phone.
- **Several people, one copy.** Each person has their own subscriptions and their own read, kept
and playback state. There is one file on disk per episode, however many people want it. People
sign in with a password or through a proxy (Cloudflare Zero Trust or Authentik), and admins
manage accounts and settings.
- **Scanning.** Feeds are checked on a schedule, globally or per feed, and a feed's own TTL is
honoured. Keyword, explicit-content and media-type filters decide what is downloaded, with a cap
on new downloads per scan.
- **Downloads.** Files come over HTTP or BitTorrent and are filed into a folder per feed.
Retention deletes the oldest files to stay under a disk quota or an age limit, and never touches
an item someone has kept.
- **OPML.** You can import and export your own subscriptions. You can also subscribe to an OPML
URL, which keeps a whole list in step as a folder.
What does not: iTunes and iPhoto export via AppleScript, text-to-speech enclosures, the Windows
WMP/COM paths, XML plists and Python pickles for state, the `directory.iPodderX.com` survey ping,
3DES-encrypted preferences, and the `printMSG` stdout protocol -- replaced by a JSON-lines socket.
## Run it
## Quick start
With Docker:
```sh
docker build -t ipodderx .
docker compose up -d
```
`docker-compose.yml` is set up for the author's own server. Point its `image` and its three volumes
(`/config`, `/data` and `/downloads`) at yours first. The UI is on port 8099. BitTorrent uses 6881
over TCP and UDP. Files are written as `PUID`/`PGID`, 99:100 by default.
From source:
```sh
cargo build --release
install -m755 target/release/ipx ~/.cargo/bin/
ipx add https://atp.fm/rss # subscribe
ipx fetch # scan and download
ipx daemon # scheduler, control socket and web UI
./target/release/ipx daemon
```
On first start with `[web] enabled = true` the daemon mints a token, writes it to config.toml and
prints the URL to open. A database with no accounts starts with **admin / ipodderx** at `/login` --
change it with `echo -n '<password>' | ipx user passwd admin`.
The first start creates **admin / ipodderx**. Sign in at `/login`, then change it:
```sh
echo -n 'a good password' | ipx user passwd admin
```
The UI is plain HTTP, so put TLS in front of it if it is reachable from outside your network.
## Documentation
| | |
|---|---|
| [docs/configuration.md](docs/configuration.md) | Every config key, paths, environment variables |
| [docs/configuration.md](docs/configuration.md) | Every config key, path and environment variable |
| [docs/cli.md](docs/cli.md) | Every command, including `ipx user` |
| [docs/users.md](docs/users.md) | Accounts, and what several people share |
| [docs/sso.md](docs/sso.md) | Cloudflare Zero Trust or Authentik in front |
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, socket, HTTP API |
| [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik |
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API |
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
| [docs/history.md](docs/history.md) | How it was built: the long form, with what was wrong and why |
| [CLAUDE.md](CLAUDE.md) | Notes for anyone (or anything) working on the code |
## The web UI
`ipx daemon` serves it in the same process, so it reads SQLite and the event bus directly.
Feeds down the side; the selected feed's items across the top; the selected item's text and its
enclosures below, which is where you play, download or delete them. The divider drags and its
position is remembered. Playback serves Range requests, so seeking works. An OPML subscription is a
collapsible folder whose page lists the feeds inside it.
An item may carry several enclosures; all of them appear below, and anything that is not audio or
video gets a View link rather than a player -- the publisher's copy until it is downloaded, the
local one after. Opening an item marks it read. Show notes are untrusted feed HTML, sanitized with
`ammonia` server-side before they reach the page.
The **Log** button shows the running daemon live in four tabs: *Daemon I/O* is the control protocol
itself, every command in and event out; *Scans* is feed and download activity; *HTTP* is web
requests; *All* is everything, with level and text filters and a copy button. It reads a ring buffer
held in the process, not a file, so it works the same under Docker.
It is plain HTTP. On a LAN bind the token and everything else cross the network in the clear, and a
feed URL can itself carry a credential. Put TLS in front of it if that matters.
## OPML
**Importing and exporting** a file copies subscriptions in or out once: `ipx import subs.opml`,
`ipx export subs.opml`, or Settings → Subscriptions in the UI.
**Subscribing to an OPML URL** is a live subscription, as iPodderX had. Add the OPML's URL like any
other feed; every scan re-reads it and keeps your list in step. The feeds inside are not written to
config.toml -- the OPML is the source of truth, so they are re-derived each scan and held in the
database. They show as a folder, download into one nested folder, and inherit the subscription's
settings until you change one, which gives it its own entry.
When a feed drops out of the OPML upstream, it is unsubscribed and removed -- unless it has
downloads, in which case it is kept and flagged in the UI as no longer listed. A downloaded file is
never left behind with nothing explaining where it came from.
## Docker
```sh
docker buildx build --tag 192.168.1.130:5000/ipodderx:latest . --push
docker compose pull ipodderx && docker compose up -d ipodderx
docker compose logs -f ipodderx # the first start prints the default admin password
```
`docker-compose.yml` runs the image from the registry above rather than building it, so build and
push first; change the tag in both places to use another registry. It mounts `/config` (config.toml),
`/data` (state.db) and `/downloads` from this install's host paths, which you will want to change for
yours. It publishes 8099 for the
UI and 6881 (TCP **and** UDP -- DHT needs the UDP side), and sets `PUID`/`PGID` to `99:100` so files
land owned the way Unraid shares expect. The healthcheck runs `ipx status` through the control
socket, so it catches a daemon that is alive but wedged rather than merely one that has died.
## Running it as a service
`contrib/` has a systemd user unit for the daemon, and a timer plus one-shot service if you would
rather run periodic scans with no daemon -- in which case there is no socket for a UI to attach to.
| [docs/history.md](docs/history.md) | How it was built, with what was wrong and why |
| [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed |
## Tests
```sh
cargo test # the engine: parsing, filters, retention, schedules, SQL, per-user state
node tests/page-smoke.js # the page script loads without throwing
npx playwright test # a real browser against a real daemon
npx playwright test # a real browser against a real daemon on fixture feeds
```
`npm install` gets the test runner; the browser comes from
`npx playwright install --with-deps chromium` (in `install.sh`).
`npm install` gets the test runner, and `npx playwright install --with-deps chromium` gets the
browser.
## License
MIT. See [LICENSE](LICENSE).
The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by @fontawesome, under
[CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG in `web/index.html`.
MIT, see [LICENSE](LICENSE). The icons are [Font Awesome Free](https://fontawesome.com) 7.3.1 by
@fontawesome, under [CC BY 4.0](https://fontawesome.com/license/free), embedded as SVG.

57
TODO.md Normal file
View File

@@ -0,0 +1,57 @@
# To do
## Errors in the log
From the production log and the feeds' stored errors on 2026-09-13. The Docker log only reached back
to 12:32 UTC, so the list comes from `feeds.last_error`: 57 of 1,059 feeds. None of it is ipx's
User-Agent; a browser gets the same answers.
- [x] **Stop scanning an OPML's feeds once nobody subscribes to it.** 55 of the 57 are feeds from
`davewiner` (lists.opml.org/davefeeds.xml). The list left `config.toml` about 14 hours before this
was written, but its 922 feeds are still in the database and still scanned every hour, with
auto-download on: `subscriptions()` adds every derived feed, and with no parent to copy from,
`auto_download` falls back to `true` (`src/main.rs:937`). `docs/users.md` says a feed stops being
scanned when its last subscriber leaves. Skip a derived feed whose parent is not in config, and
when the last subscriber leaves an OPML, treat its feeds the way `sync_group` treats ones the list
dropped: remove those with nothing downloaded, mark the rest orphaned. `remove_feed` and `ipx rm`
both leave them behind today. (`src/main.rs`, `src/web.rs`)
- [x] **Read feeds with a bare `&`.** kcpw has `<link>https://kcpw.org/?post_type=post&p=125715</link>`,
and both feedland feeds have the same fault. Strict XML refuses them; browsers and other readers do
not. When `feed::parse` fails, try once more with every `&` that does not start an entity written
as `&amp;`. Nobody subscribes to these three now, but the next feed like them will fail the same
way. (`src/feed.rs`)
- [x] **Say what came back when it is not a feed.** Thirteen errors read "not RSS (the input did not
begin with an rss tag) and not Atom (...)". Each one checked was a web page: the feed moved and its
old URL redirects to the site, or the domain lapsed. Say "got a web page, not a feed", and when the
page links a feed (`<link rel="alternate" type="application/rss+xml">`), name it. That link found
the new feed for om.co, ms.now, Letters of Note, the Daily Dot, Hell Gate, The Frame Lab and
Daily Kos. A `202` with an empty body (British Antarctic Survey) should read as "nothing yet", not
as a parse failure. (`src/feed.rs`)
- [x] **Show a publisher's error in the UI.** Today a failing feed shows its raw error in red only
once you open it (`web/index.html:877`, `:960`); the OPML view marks a failing child "error"
(`:988`), and the sidebar shows nothing. Mark a failing feed in the sidebar too, and say whose
problem it is and what to do, in plain words: a 404 means the publisher took the feed down or moved
it, 401 and 403 that the site refuses ipx, 402 that the feed needs a paid plan, a name that no
longer resolves that the site is gone, and a web page instead of a feed that it moved (with the new
URL when the page links one). Offer Unsubscribe, or Use the new address. Keep timeouts, 5xx and
429 out of it, and flag a feed only once it has failed for a day: macmanx failed once and read
fine an hour later. That needs the time a feed started failing, kept beside `last_error` and
cleared when a check succeeds. (`web/index.html`, `src/db.rs`, `src/main.rs`)
- [ ] **Westenberg.** The only failing feed anyone subscribes to. The Hacker News stars list still
gives `joanwestenberg.com/rss`, which is a 404; the feed is now `joanwestenberg.com/feed`.
Nothing for ipx to fix; subscribe to the new URL directly.
## Other Fixes and Features
- [ ] Remember which feed is selected and view (all, unread, flagged, etc) user as selected between visits. If unknown default to All Subscriptions
- [x] When clicking any link it should open in a new tab
- [ ] In mobile (iOS) sometimes the top line items like the hamburger menu are not clickable unless you do a hard refresh
- [x] video files play as audio files, they should play as video.
- [ ] Move Light/Dark/Classic options to user settings. Include an Auto mode that uses system preferences for light/dark modes
- [ ] Below Popular, have a currently listening section to show what podcasts have been started and not finnished
- [x] Update subscribe/unsubscribe icons to be circle-minus (unsubscribe) and circle-check (subscribe)
- [x] If I'm on the Unread tab, and I click to read an item the entry in the list will disappear. it should remain until I click to another item.
## Directory Overhaul
- [ ] Directory needs to be more functional, with categories and a more interesting layout. use /frontend-design to help

View File

@@ -1,7 +0,0 @@
[Unit]
Description=ipx feed scan (one shot)
[Service]
Type=oneshot
ExecStart=%h/.cargo/bin/ipx fetch
Environment=IPX_LOG=ipx=info

View File

@@ -1,18 +0,0 @@
# User unit: install to ~/.config/systemd/user/ipx.service, then
# systemctl --user enable --now ipx
# The socket lands in $XDG_RUNTIME_DIR/ipx.sock by default, so a UI running as the
# same user can attach without extra configuration.
[Unit]
Description=ipx podcatcher
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=%h/.cargo/bin/ipx daemon
Restart=on-failure
RestartSec=30
Environment=IPX_LOG=ipx=info
[Install]
WantedBy=default.target

View File

@@ -1,16 +0,0 @@
# Alternative to the daemon: a periodic one-shot scan, closer to how the original
# iPodderX agent was driven. Use this OR ipx.service, not both -- with no daemon
# running there is no socket, so a UI cannot attach.
#
# Install ipx-scan.service and ipx.timer to ~/.config/systemd/user/, then
# systemctl --user enable --now ipx.timer
[Unit]
Description=Periodic ipx feed scan
[Timer]
OnBootSec=5min
OnUnitActiveSec=1h
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -28,12 +28,15 @@ The page is compiled in, so **editing `web/index.html` needs a rebuild**.
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
re-derived into the database each scan, never written to config.toml.
re-derived into the database each scan, never written to config.toml. A Patreon creator link
(a token, no `show=`) with more than one show is treated the same way, before any fetch: its
shows come from Patreon's web API and each becomes a derived feed.
4. Record entries. A changed title or description flips the item back to unread.
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
a reason.
a reason. What a filter skipped is judged again every scan, so a change of settings brings it
back. A feed in a group takes your settings on the group for anything you have not set on it.
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
or an error page — and is deleted.
@@ -47,21 +50,23 @@ entries feed_id, guid, title, link, published, description, first_seen,
image, duration, episode, season PK (feed_id, guid)
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,
bytes_done, downloaded_at, last_error
users id, name, pass_hash, is_admin, created
sessions token, user_id, created, seen
users id, name, pass_hash, is_admin, created, last_login
sessions token, user_id, seen
subscriptions user_id, feed_id, keywords, auto_download, allow_explicit,
max_new_per_check, created PK (user_id, feed_id)
max_new_per_check PK (user_id, feed_id)
entry_state user_id, feed_id, guid, read, flagged, position
PK (user_id, feed_id, guid)
```
`entries` still has `read`, `flagged` and `position` columns from before accounts existed. They are
**dead** — the migration copied them into `entry_state` and nothing reads them now. Anything found
querying them is a bug; two were.
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
older database.
Schema changes: add the table or column to `SCHEMA`, and for a column also to the list in
`migrate()`, which does `PRAGMA table_info` then `ALTER TABLE ADD COLUMN`. `Db::memory()` runs the
same path as `Db::open`, so a migration-only column cannot pass tests while missing in production.
Schema changes: add the table or column to `SCHEMA`. `CREATE TABLE IF NOT EXISTS` leaves a table
that already exists alone, so a new column on one also goes in `migrate()`'s `wanted` list, and a
retired one in its `retired` list; both are checked with `PRAGMA table_info`. Columns from before
0.3.0, the oldest version an upgrade may start from, need no entry. `Db::memory()` runs the same
path as `Db::open`, so a migration cannot pass the tests while missing in production.
## Control socket
@@ -82,7 +87,9 @@ printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/
**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
there.
there. Commands run one at a time, in the order they arrive, except `status`: the socket answers it
straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and
answers only the client that asked, since `status` would end any other client's session.
Progress carries the enclosure id, without which a UI cannot tell one download from another and
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
@@ -108,11 +115,11 @@ else a `401`.
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, starred, position |
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, kept, position |
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
| `POST /api/fetch` | |
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with everyone counted (id, title, art, count, whether it is yours; never a URL, never a private feed); subscribe by id |
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with an OPML's feeds in place of the OPML and everyone counted (id, title, art, count, whether it is yours; never a URL, never a private feed); subscribe by id |
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |
| `GET /api/events` | SSE, the same broadcast the socket carries |

View File

@@ -60,7 +60,7 @@ ipx reap # actually delete
```
Files are deleted to get back under `max_total_gb`, oldest first, and items past `max_age_days`
with no file are pruned from the database. **Starred by anyone keeps a file**, and one only counts
with no file are pruned from the database. **An item anyone kept keeps its file**, and one only counts
as read when everyone subscribed has read it. The enclosure row survives as `reaped`, which is what
stops the next scan fetching it again.

View File

@@ -32,10 +32,10 @@ media_types = ["audio", "video"]
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
pad. Starred items are never deleted, and a file only counts as read once every subscriber has
pad. Kept items are never deleted, and a file only counts as read once every subscriber has
read it. `0` disables it entirely.
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
Starred ones stay. `0` disables it.
Kept ones stay. `0` disables it.
* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue.
`0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap
fetched 216 files and 22 GB in one scan.
@@ -43,8 +43,6 @@ media_types = ["audio", "video"]
can be fetched by hand; blog feeds put each article's header image in an `<enclosure>`, and
without this the disk fills with artwork. Empty takes everything.
`interval_mins` from older configs is still read, and `schedule` supersedes it.
## `[torrent]`
```toml
@@ -69,6 +67,7 @@ token = "" # generated and saved on first run
trusted_header = "" # e.g. "Cf-Access-Authenticated-User-Email"
trusted_proxies = ["127.0.0.1", "::1"]
auto_create_users = true
sign_out_url = "" # e.g. "/cdn-cgi/access/logout"
session_days = 30
```
@@ -79,6 +78,9 @@ session_days = 30
* **`trusted_proxies`** — addresses allowed to assert that header, and the entire security boundary
for it. Name the proxy, never a subnet.
* **`auto_create_users`** — create an account the first time the proxy vouches for a new name.
* **`sign_out_url`** — where Sign out sends someone the proxy signed in: the proxy's own sign-out,
`/cdn-cgi/access/logout` behind Cloudflare Access. Empty sends them to the sign-in page, where
the proxy signs them straight back in.
* **`session_days`** — sign a session out after this long without a request.
It is plain HTTP. On a LAN bind everything crosses the network in the clear — and a feed URL can

View File

@@ -6,6 +6,257 @@ reasoning lives. New write-ups go at the top.
See [README.md](../README.md) for what the thing is.
## 2026-09-14 — Settings, for everyone with an account
A user reported that Settings disappeared shortly after they signed in: it showed for a moment,
then was gone. `#prefs` sat inside the same `.tgroup` as `#logs`, and `api('/api/me')` hid the
whole group -- `$('#admintools').hidden=true` -- the moment it learned the account was not an
admin. Nothing wrong with that check timing; it was hiding the wrong thing.
The Settings modal is not actually all-or-nothing. `GET /api/settings`, and Export and Import
OPML, carry no admin check server-side -- `export_opml` and `import_opml` work from a user's own
subscriptions, and the schedule/quota page is read-only information, not a control. Only the
`PATCH` that changes those settings, and the Users screen behind it, return 403 for anyone but an
admin. The comment above the old hide -- "scanning, quotas, accounts and the log are the
operator's business" -- was wrong about quotas and half wrong about accounts: reading them is
everyone's; changing them is the operator's.
`prefsModal()` now branches on `S.me.admin` the way the per-feed settings modal already does for
its URL field: a non-admin gets the schedule and quota as text, Subscriptions (Export/Import)
in full, and no Users section or Save button. Only `#logs` stays hidden, since the log names every
account and every failed sign-in. The browser test for a second account asserted the old
behaviour outright (`#prefs` hidden, not an admin) rather than what the server actually allows;
fixing the UI meant fixing the test's premise too, not just the assertion.
## 2026-09-12 — Healthy while busy
After a deploy the container sat at "starting" for a minute, and Docker's health log showed two
`ipx status` probes exceeding their 5-second timeout. The daemon's own log explained it. The first
scan after the start fetched 23 feeds, from 14:10:41 to 14:11:35, and both probes' `status`
commands waited in the job queue behind it; they were answered together at 14:11:35, straight after
`scan_done`. The worker runs one job at a time and `status` was one of its jobs, so any scan or
download longer than about a minute and a half, three 30-second probes, would have had Docker call
a working daemon unhealthy.
The socket now answers `status` itself, from two short queries, and only real work goes through the
queue. The trade is that healthy now means the daemon answers on its socket and can read its
database; a worker stuck on one job would still pass. Asking a daemon that downloads hour-long
podcasts to be idle within five seconds was never a fair test of whether it was alive. A test holds
the queue full and checks `status` still comes back.
The first version broadcast the answer, as the queued one had been. Timing `status` during a forced
scan in production showed the catch: `status` is a terminal event, so the `ipx fetch` watching that
scan stopped reading at the first probe and printed the status line as its last, while the scan
carried on. When `status` waited behind the scan it could never arrive first, so this had never
shown. The answer now goes only to the client that asked, and the test checks that another client
hears nothing.
## 2026-09-12 — Signing in through Authentik, for real
Ray could not get Authentik's sign-in to reach ipx, following `docs/sso.md`, which had been written
without ever being tried. Looking at the Cloudflare account through its API showed that side was
already complete. Authentik is Zero Trust's OpenID Connect identity provider; the Access application
`ipodderx` allows only it and a list of five addresses; the tunnel `rays-unraid` routes
`ipodderx.sdf1.net` to `192.168.1.130:8099`; DNS is a proxied CNAME to the tunnel. Access's log
showed `rays@sdf1.net` signing in through it. Nothing on Cloudflare was changed, so no other site
was touched.
The gaps were all at ipx's end: `trusted_header` was empty, `trusted_proxies` held only loopback,
and the account was called `rays` while the header carries `rays@sdf1.net`.
Finding the address to trust took the most time. The page said `127.0.0.1`, but `cloudflared` runs in
its own container and reaches ipx through the host's published port. ipx logs no peer addresses, so
the address was read from `/proc/net/tcp` inside the ipx container: `192.168.16.1`, the gateway of
`content_default`, where Docker's masquerade puts traffic crossing from another bridge. A request
from Tower's own shell arrived as `192.168.1.130` instead, and a throwaway `busybox` on the default
bridge as `192.168.16.1`: the first was refused with the header, the second believed. LAN machines
keep their own addresses, since Docker forwards published ports with iptables (the userland proxy
only handles loopback).
Every change, in order, with how to undo it:
1. **Code**, commit `586d2c0`: `ipx user rename`, deployed. Revert the commit and redeploy to
remove it; nothing depends on it once used.
2. **Account**: `docker exec iPodderX ipx user rename rays rays@sdf1.net`. Same id, so its feeds,
read state, password and admin rights stayed. Undo: `docker exec iPodderX ipx user rename
rays@sdf1.net rays`. Signing in at `/login` now takes the new name.
3. **Config**, `/mnt/fast/appdata/ipodderx/config.toml`, `[web]`: `trusted_header` from `""` to
`"Cf-Access-Authenticated-User-Email"`, and `"192.168.16.1"` added to `trusted_proxies`. The
file as it was is `config.toml.2026-09-12-sso.bak` beside it. Undo: copy the backup back and
`docker compose -f /mnt/fast/arcane/projects/content/compose.yaml restart ipodderx`.
4. **Cloudflare, Docker networks and other containers**: unchanged. The `busybox` test container
was removed when it exited, and its image afterwards.
5. **Authentik**, later the same day, because ipodderx had no tile in its library while Outline
did: a bookmark application `ipodderx` (pk `5854a98e-816a-4c4f-9f27-63e69dc29d1d`), made
through the API with a token of Ray's. No provider and no policy bindings, like Outline's, the
iPodderX icon, and a link to `https://ipodderx.sdf1.net`. It changes nothing about who can sign
in. Undo: delete it under Applications → Applications, or
`DELETE /api/v3/core/applications/ipodderx/`.
6. **Signing out**, later again. Sign out landed on ipx's password page while Access still vouched
for Ray, so it signed nothing out, and the page looked like the wrong login. Cloudflare's
`/cdn-cgi/access/logout` ends the Access session for every Access application at once (there is
no per-application sign-out, and it takes no redirect), and Authentik's end-session only ends
one application's session unless single logout is set up there. Ray chose Access's sign-out. New
`[web] sign_out_url`, set to `/cdn-cgi/access/logout` in production (the file as it was is
`config.toml.2026-09-12-signout.bak`), and `/login` now sends anyone the proxy vouches for on to
`/`. Undo: take the key out and restart; the code does nothing without it.
What the address trusts is any container on Tower that connects through the host's port, not only
`cloudflared`. Verifying Cloudflare's signed `Cf-Access-Jwt-Assertion` would remove that, and is
the upgrade if it matters.
## 2026-09-12 — Trimming the state database
An audit of the database layer, with a read-only copy of production to check it against. The
data was already clean: no tables or indexes left from older versions, 47 free pages after the
column drops earlier the same day, and one stray `entry_state` row. The code had five things:
- `migrate()` still added eight columns to any table missing them. All eight shipped in 0.2.0 and
upgrades now start from 0.3.0 at the oldest, so the list and its loop went; the `retired` drop
list stays, since a database coming from 0.4.0 still has the old read columns.
- `created` on `users`, `subscriptions` and `sessions` was written by every insert and read by
nothing. They joined `retired`. The old-database test now builds all three tables, foreign keys
included, since `DROP COLUMN` on a table that references another was the part worth proving.
- `Db::subscribed_feed_ids` had no callers, though its doc said the scanner walked it.
`Db::subscriber_count` had one caller asking whether it was above zero, which
`subscriber_counts().contains_key` answers. `Managed.orphaned` was selected and never read.
- `users.created` came back the same afternoon, with `last_login` beside it. Nothing read it, but
when an account was made and when it last signed in is what you want to know when tidying
accounts, and it cannot be recovered later. Both existing accounts got their creation times back
from the backup taken before the drop, and a last sign-in from their newest session in it.
`last_login` is kept to the hour, because the proxy vouches for every request and that would
otherwise be a write each time.
## 2026-09-12 — Cutting what had outlived its reason
A whole-repo audit for over-engineering listed twelve things to cut, and all of them went.
- **Upgrades from before accounts.** `migrate_opml_children` moved OPML feeds that old versions
wrote into `config.toml` out to the database, and ran at every daemon start to do nothing after
the first. Production ran it in 0.3.0; anything older has to pass through 0.4.0.
- **Half of the adoption, and not the other half.** The audit called `adopt_existing_library` a
one-time migration and it was cut whole. It did two jobs: copy the old read state into
`entry_state`, which was dead, and subscribe the first admin to the whole catalogue while nobody
subscribed to anything, which is how a fresh install's first account gets `config.toml`'s feeds.
The browser suite caught it at once, signing in to an empty sidebar; `cargo test` had no idea.
The second job is back as `adopt_catalogue`, with a unit test of its own.
- **The dead `entries` columns.** `read`, `flagged` and `position` moved to `entry_state` with
accounts. The adoption's copy was their last reader, but `record_entry` still wrote them, and
still reset `read` when a title changed, which nothing looked at. Two bugs came from queries
reading them. `migrate()` now drops them from an existing database (SQLite has had `DROP COLUMN`
since 3.35), and a test builds an old table to prove it. On production each drop rewrote the
66 MB `entries` table, about four seconds apiece, so the first start took thirteen. An
`ipx status` run in that window failed with `migrating schema`: every `ipx` command migrates when
it opens the database, and it collided with the daemon doing the same. A failed `ALTER TABLE`
changes nothing, and the database had been copied to `backup/` first anyway.
- **`interval_mins`**, which `schedule` replaced. An old config that still has the key loads; the
key is ignored, and the config test carries it to keep that true.
- **Three dependencies.** `infer` was only asked whether a file is a torrent, and the check after
it already looked for `d8:announce`, which is what `infer` looks for. `dirs` was three lookups of
`XDG_CONFIG_HOME`, `XDG_DATA_HOME` and `HOME`. `tokio-stream` wrapped the broadcast receiver for
the event stream; `futures_util::stream::unfold` does the same, lagging clients included.
- **Two token generators.** The web token came from a copy of the session-token code, with a
clock fallback on top. It uses `auth::new_session_token` now, and is 64 characters.
- **The icon inlined four times**, 23 KB of base64 each, into both pages. It is `/icon.png` now,
outside the sign-in wall with `/login`, since the sign-in page shows it.
- Also: the `contrib/` systemd units from before the container, `Db::entries` and
`Db::count_entries` that only the tests called, three `logbuf` visitors that repeated the trait's
defaults, and unused state, a helper and dead CSS in the page.
## 2026-09-11 — A design pass on the web UI
A review against screenshots of every view in all three themes found that Dark and Light read as a
generic dark dashboard: one pale blue did every job, most labels were bold, and nothing led. The
list it produced, in `TODO.md`, was worked through in one go. What was worth knowing:
- **Amber means new.** Badges, unread dots and download bars take the icon's EQ amber; blue is left
for the primary action and links. Light's amber was `#b06f10`, which gives white text 4.1:1,
short of AA for 11 px bold. It is `#9a5f0a` now, 5.2:1.
- **EQ bars mark what is playing.** Three `<i>` bars stand at 60, 100 and 40 % and animate only
while `body.playing` is set. The first version left the animation on but paused, expecting each
bar to hold a different frame. The frames it held were within a pixel of each other, and on
screen the bars read as three dots. Under reduced motion one rule drops every animation and
transition, which leaves the bars standing.
- **The focus ring was clipped.** `.tgroup` and `#topbar` both set `overflow:hidden`, so a ring
drawn outside a toolbar button was cut off. Rings inside clipping parents are inset instead.
- **The feed list could not be used from the keyboard at all.** Rows were `<div>`s and the triangle
a `<span>`, so Tab went from the feed filter to Sign out. Rows now take focus, the triangle is a
`<button aria-expanded>`, and `renderFeeds` puts focus back on the same row after redrawing,
since every live update replaces every row. The list's own key handler stops Space and the
arrows from reaching the player's shortcuts on the document.
- **The triangle hangs in the margin.** Every row used to reserve an 18 px slot for it, pushing a
hundred feeds 28 px right of the places above for the sake of two folders. It is now absolutely
placed in the row's left padding, the full height of the row, so a near miss no longer opens
the folder's page.
- **A selected tile vanished** because the initials tile and the selected row were both `--raise`.
Tiles now mix their tint into `--bg`, which no row uses.
- **The Files pane hides itself** with `#split:has(>#files[hidden])`, which collapses its column.
The phone layout already hides the pane, and a zero-width extra track there is harmless.
- **Flagged became Kept** in the tab, and in the server's refusal to delete a file someone else
kept. The filter value and the column stay `flagged`; renaming those buys nothing.
## 2026-09-11 — A Patreon creator is a list of shows
Ray asked whether ipx could sync with Patreon. Not in full. The documented API (v2, the
`identity.memberships` scope) lists the creators you back and whether each has a feed (`has_rss`),
but no resource carries the `auth` token that makes a feed URL work. That token only comes from the
creator's page. It is also one per membership, not one per account: techpod's differs from Glass
Cannon's, so no single token finds everything you back.
What does work is one creator at a time, which is what Ray wanted for Glass Cannon and its 33 shows:
- `patreon.com/rss?auth=<token>`, with no creator named, returns that token's creator. Its self link,
about 660 bytes in, gives the campaign by number (`/rss/369921`). Patreon ignores `Range` here, so
ipx reads the stream until the number appears and hangs up, instead of taking all 2.8 MB.
- A show's `show=` number is a Patreon collection. Asked anonymously, the collection listing
(`/api/collection?filter[campaign_id]=`) and a post's `collections` both hide patron-only ones: you
get "FAQ". `/api/campaigns/<id>?include=shows` lists every show, anonymously, in one response.
- Every spelling works: `rss/glasscannon?auth=…&show=N`, `rss/369921?…` and `rss?auth=…&show=N` all
serve the same 131 items. Enclosure URLs are the same in the creator feed and the show feed, and
stable between fetches.
That last point shaped the design. `enclosures.url` is unique, so whichever feed is scanned first owns
the file. The first cut only asked a creator for its shows while it had no entries of its own, so that a
creator already read as a plain feed, holding every show's episodes, would never be split into shows
that came up empty. Within the hour that was the wrong call: Glass Cannon had gone into production on
the build before this one, been read as one feed of 2,385 items, and the rule kept it that way. Finding
anything in that heap was the problem Ray wanted solved.
So a creator with more than one show is always a group, run through the same sync as an OPML
(`sync_group`, split out of `sync_opml`). When it becomes one, its items are cleared and each show
takes over the enclosures the creator holds as the show lists them (`Db::adopt`), downloaded files and
everyone's read state included. One show leaves it a plain feed, which is what techpod already was. If
the shows cannot be listed, a creator already split fails the scan rather than being read as one heap;
one that never was is read as one feed until they can be. An answer without a `shows` list is an error,
not "no shows".
**Filter verdicts follow the settings.** Ray also reported that turning on Allow explicit and
rescanning brought nothing back. An item was judged once, when first seen, and `skipped` was final. The
2026-09-10 entry below saw it coming ("worth a `ipx retry <feed>` command if this bites"). It bit:
2,166 Glass Cannon items and all 88 of Shadowdark's sat at `skipped: explicit` with the setting on.
Every scan now runs the filters again over what they skipped (not over `torrents disabled`, which is not
a filter's call) and requeues what they now let through. Only that direction: a queued item is never
pulled back, because Download latest and a manual download both work by queueing.
**Two gaps beside it.** Add feed had no explicit box, so every new feed's first scan skipped all its
explicit items; it has one now, stored on your subscription like the feed dialog's. And a feed inside a
group ignored your settings on the group, though the group's dialog said they were inherited: settings
live on each person's subscription, and nothing read the group's. `Db::subscribers` now fills what you
have not set on the feed from your subscription to the group, and the feed list shows the same.
**A name that was already used.** Replaying the split on a copy of the production database left one
show with a Supercast episode in it. "Glass Cannon Live! Ascension | Pathfinder 2E" slugs to
`glass-cannon-live-ascension-pathfinder-2`, the id of a Supercast feed of the same show that had been
removed. Removing a feed keeps its rows on purpose, so that re-adding it does not fetch the back
catalogue again, but choosing a new id only checked config.toml and derived feeds. The Patreon show took
the old id and everything still filed under it. An id is now also taken when the database has a feed by
that id at a different URL; the same URL may still have it back, which is the re-add case.
Shows already added by hand are matched by token and show number, not by exact URL (`same_feed`), so a
bare token does not add Get in the Trunk and Shadowdark a second time under another spelling.
The show listing is Patreon's own undocumented web API. If it changes, only finding new shows stops.
## 2026-09-11 — One meaning per icon, sortable columns, and one player
Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act

View File

@@ -1,9 +1,8 @@
# Signing in through Cloudflare Zero Trust or Authentik
# Signing in through Cloudflare Access and Authentik
ipx can take the signed-in identity from whatever sits in front of it, instead of asking for a
password itself. Both products below do the same thing in the end: they authenticate the person and
pass the result to the origin in a **header**. ipx reads that header, finds (or creates) the
matching account, and gets on with it.
password itself. The proxy authenticates the person and passes the result to ipx in a **header**;
ipx reads it, finds (or creates) the matching account, and gets on with it.
Read [How this is secured](#how-this-is-secured) before exposing anything. The short version: a
header is worth exactly as much as the hop that set it, so ipx only believes one from an address you
@@ -11,195 +10,173 @@ list.
---
## The ipx side (both setups)
## How ipodderx.sdf1.net does it
Checked end to end on 2026-09-12. An earlier version of this page had never been tried against a
real setup and pointed at the wrong address.
```
browser ─► Cloudflare Access, app "ipodderx" ─── sign in ───► Authentik (OpenID Connect)
─► tunnel "rays-unraid" (the cloudflared container on Tower)
─► http://192.168.1.130:8099 ─► ipx
```
Authentik is not in the request path. It is the identity provider Cloudflare Access asks. Access
then adds `Cf-Access-Authenticated-User-Email`, the email address Authentik gave it, to every
request it forwards through the tunnel, and ipx signs that person in.
| Piece | Where | Setting |
|---|---|---|
| Identity provider | Zero Trust → Settings → Authentication | `Authentik`, OpenID Connect; scopes `openid email profile` |
| Access application | Zero Trust → Access → Applications → `ipodderx` | Domain `ipodderx.sdf1.net`; identity providers: Authentik only, with instant auth; session 730h; policy *Require Login* allows a list of email addresses |
| Tunnel route | Zero Trust → Networks → Tunnels → `rays-unraid` → Public hostnames | `ipodderx.sdf1.net` → HTTP `192.168.1.130:8099` |
| DNS | `sdf1.net` | `ipodderx` CNAME to the tunnel, proxied |
| ipx | `/mnt/fast/appdata/ipodderx/config.toml`, `[web]` | below |
```toml
[web]
enabled = true
bind = "0.0.0.0:8099"
token = "…" # keep it: it is the admin, used by the healthcheck
# The header your proxy sets. Empty (the default) disables this whole path.
trusted_header = "Cf-Access-Authenticated-User-Email" # Authentik: "X-authentik-username"
# Addresses allowed to assert that header -- the proxy, and nothing else.
trusted_proxies = ["127.0.0.1", "::1"]
# Create an account the first time the proxy vouches for a name ipx has not seen.
bind = "0.0.0.0:8099"
trusted_header = "Cf-Access-Authenticated-User-Email"
trusted_proxies = ["127.0.0.1", "::1", "192.168.16.1"]
auto_create_users = true
sign_out_url = "/cdn-cgi/access/logout"
session_days = 30
```
Restart the daemon after editing. Accounts made this way have **no password**: they can only ever
arrive through the proxy. `ipx user list` marks them `proxy only`.
Restart ipx after editing it: `docker compose -f /mnt/fast/arcane/projects/content/compose.yaml
restart ipodderx`.
The first account created is an admin. Every later one is an ordinary user, and an ordinary user
cannot change global settings, a feed's URL or folder, or how often feeds are scanned: the API
refuses those with a `403`, not just the UI. Everything else about a feed (which items they want,
whether to fetch them, how many at a time) is theirs alone; see [users.md](users.md).
### What was missing
Somebody arriving through the proxy for the first time starts with **no feeds**, because
subscriptions are per person. Adding a feed someone else already reads costs no second fetch and no
second copy on disk.
Cloudflare and Authentik were already right. Three things on the ipx side were not:
Promote someone with:
1. **`trusted_header` was empty**, which switches the whole proxy path off. ipx ignored the header
and asked for a password.
2. **`trusted_proxies` listed only `127.0.0.1`.** The tunnel's requests do not come from there;
see the next section.
3. **The account had the wrong name.** It was made by hand as `rays`, but the header carries
`rays@sdf1.net`. With `auto_create_users` on, the first visit would have made a second, empty
account. `ipx user rename rays rays@sdf1.net` fixed that without losing anything.
### The address to trust, and why it is 192.168.16.1
`cloudflared` runs in its own container and reaches ipx through the host's published port. Docker
(iptables firewall backend) masquerades traffic between its bridge networks, so the tunnel's
requests arrive from the **gateway of ipx's own network**, `content_default`:
```sh
ipx user list
echo -n 'a good password' | ipx user passwd <name> # optional: also lets them sign in directly
docker network inspect content_default -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'
```
Local sign-in at `/login` keeps working alongside all of this, which is how you get in from the LAN
when the tunnel is down. So does the shared `[web] token`, which signs in as the admin: that is
what the Docker healthcheck uses, and the way back in if you lock yourself out. A brand new database
starts with **admin / ipodderx** — change it.
That was measured, not assumed. ipx does not log where a request came from, so the addresses were
read from the kernel's connection table inside the container while the site was open. (`/proc/net/tcp`
lists them in hex.)
If the `content` project's network is ever recreated, its gateway can change. Check it again, and
update `trusted_proxies` to match.
### Names
The username is the email address, lower-cased: `rays@sdf1.net`. To sign in at `/login` with a
password from the LAN, use that name too.
To let someone else in, add their address to the Access policy; they need an Authentik account with
that email. With `auto_create_users = true` they get an ipx account on their first visit, as an
ordinary user with no feeds. An account made before the proxy can be given the name the proxy will
send:
```sh
docker exec iPodderX ipx user rename <old name> <email address>
```
### Signing out
**Sign out** sends someone the proxy signed in to `sign_out_url`, here Cloudflare's
`/cdn-cgi/access/logout`. That ends your Access session for **every** Access application,
`code.sdf1.net` included: Cloudflare has no way to end just one, and its sign-out page does not send
you anywhere afterwards. The next visit goes back through Authentik, which lets you straight in if
you are still signed in there. Signing out of Authentik itself is Authentik's own sign-out.
ipx never shows its password page to someone the proxy vouches for: `/login` sends them on to their
feeds.
### The tile in Authentik's library
Authentik's library lists Authentik's own applications, and ipodderx signs in through the one
called `Cloudflare Access`, so ipodderx needs a bookmark of its own to show up there. It is
Applications → Applications → `ipodderx`: no provider, launch URL `https://ipodderx.sdf1.net`, and
the iPodderX icon. Like Outline's, it has no policy bindings, so everyone in Authentik sees the
tile. Who actually gets in is still up to the Access policy.
### Check it
```sh
# From Tower itself: not a trusted address, so the header is ignored.
curl -s -H 'Accept: application/json' -H 'Cf-Access-Authenticated-User-Email: rays@sdf1.net' \
http://192.168.1.130:8099/api/me # -> sign in
# From a container on a Docker bridge, as cloudflared is: believed.
docker run --rm --network bridge mirror.gcr.io/library/busybox wget -qO- \
--header 'Accept: application/json' --header 'Cf-Access-Authenticated-User-Email: rays@sdf1.net' \
http://192.168.1.130:8099/api/me # -> {"admin":true,"name":"rays@sdf1.net"}
```
Then open `https://ipodderx.sdf1.net` in a private window. Authentik should ask who you are, and
ipx should show `rays@sdf1.net` in the sidebar footer without asking for a password.
---
## Cloudflare Zero Trust
## The ipx settings
This is what runs `ipodderx.sdf1.net`: a `cloudflared` tunnel to the origin, with an Access
application in front of it. Cloudflare authenticates the visitor and adds
`Cf-Access-Authenticated-User-Email` to every request it forwards.
### 1. The tunnel
In **Zero Trust → Networks → Tunnels**, either use the existing tunnel or create one, then add a
public hostname:
| Field | Value |
| Key | What it does |
|---|---|
| Subdomain / domain | `ipodderx` / `sdf1.net` |
| Type | HTTP |
| URL | `localhost:8099` (or the LAN address of the box) |
| `trusted_header` | The header the proxy sets. Empty, the default, turns the proxy path off. |
| `trusted_proxies` | The addresses allowed to set it. Nothing else is believed. |
| `auto_create_users` | Make an account the first time the proxy vouches for a name ipx has not seen. |
| `sign_out_url` | Where Sign out sends someone the proxy signed in: the proxy's own sign-out. Empty sends them to the sign-in page, where the proxy signs them straight back in. |
| `session_days` | How long a password sign-in lasts without use. |
Use `localhost` when `cloudflared` runs on the same machine as ipx — that keeps the origin request
coming from `127.0.0.1`, which is already in `trusted_proxies`. If `cloudflared` runs elsewhere (its
own container, another host), put **its** address in `trusted_proxies` instead, and make sure
nothing else can reach port 8099.
The first account ever created is an admin. Every later one is an ordinary user, who cannot change
global settings, a feed's URL or folder, or how often feeds are scanned: the API refuses those with
a `403`, not just the UI. Everything else about a feed is theirs alone; see [users.md](users.md).
### 2. The Access application
**Zero Trust → Access → Applications → Add an application → Self-hosted**:
- Application domain: `ipodderx.sdf1.net`
- Session duration: whatever suits; ipx keeps its own 30-day session on top.
- Add a policy — *Allow*, with a rule such as `Emails` → your address, or `Emails ending in` →
your domain. Anyone this policy admits gets an ipx account when `auto_create_users` is on, so keep
the policy as narrow as the people you actually want reading your feeds.
### 3. Point ipx at the header
```toml
trusted_header = "Cf-Access-Authenticated-User-Email"
trusted_proxies = ["127.0.0.1", "::1"]
```
The username becomes the email address, lower-cased (`ray@example.com`). That is what shows in the
sidebar and what `ipx user list` prints.
### 4. Check it
```sh
# From the box itself: no header, no session -> the sign-in page.
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' -H 'Accept: text/html' http://127.0.0.1:8099/
# Pretending to be the tunnel (only works because 127.0.0.1 is trusted):
curl -s -H 'Cf-Access-Authenticated-User-Email: you@example.com' http://127.0.0.1:8099/api/me
```
Then load `https://ipodderx.sdf1.net` in a browser: Cloudflare should ask who you are, and ipx
should show your address in the sidebar footer without ever asking for a password.
Local sign-in at `/login` keeps working alongside the proxy, which is how you get in from the LAN
when the tunnel is down. So does the shared `[web] token`, which signs in as the admin and is the
way back in if you lock yourself out. A brand new database starts with **admin / ipodderx**;
change it.
---
## Authentik
## Authentik in the request path instead
Authentik does this with a **Proxy Provider** plus an **outpost**, which sits in the request path and
adds `X-authentik-username` (also `X-authentik-email`, `X-authentik-name`, `X-authentik-groups`).
Not what ipodderx.sdf1.net uses, and **not verified**. Authentik can also sit in front of ipx
itself, with a **Proxy Provider** and an **outpost** that adds `X-authentik-username`:
### 1. Provider
**Applications → Providers → Create → Proxy Provider**:
- Name: `ipx`
- Authorization flow: your usual (`default-provider-authorization-implicit-consent`)
- Mode: **Forward auth (single application)** if an existing reverse proxy fronts ipx, or
**Proxy** to let the outpost talk to ipx directly.
- External host: `https://ipodderx.example.net`
- Internal host (Proxy mode): `http://<ip of the ipx box>:8099`
### 2. Application and outpost
**Applications → Create**, bind it to that provider, and give it a policy so only the people you
mean are let through. Then add the provider to an outpost (**Applications → Outposts**, the embedded
one is fine).
### 3. Forward auth, if you use nginx/SWAG in front
In the server block for ipx:
```nginx
location /outpost.goauthentik.io {
proxy_pass http://authentik-server:9000/outpost.goauthentik.io;
proxy_set_header Host $host;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
add_header Set-Cookie $auth_cookie;
auth_request_set $auth_cookie $upstream_http_set_cookie;
}
location / {
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;
# This is the line that matters to ipx.
auth_request_set $authentik_username $upstream_http_x_authentik_username;
proxy_set_header X-authentik-username $authentik_username;
proxy_pass http://ipx:8099;
}
```
### 4. Point ipx at the header
```toml
trusted_header = "X-authentik-username"
trusted_proxies = ["172.18.0.5"] # the outpost or nginx container, NOT a whole subnet
```
Usernames arrive as Authentik knows them (`ray`), lower-cased.
- Applications → Providers → Create → Proxy Provider; mode **Proxy** (the outpost talks to ipx) or
**Forward auth** (an existing reverse proxy asks the outpost).
- Applications → Create, bound to that provider, with a policy; add the provider to an outpost.
- In ipx: `trusted_header = "X-authentik-username"`, and the outpost's or reverse proxy's address
in `trusted_proxies`. Measure that address as above rather than guessing it.
---
## How this is secured
**The header is only believed from `trusted_proxies`.** Every other source is ignored, and the
request falls through to a session cookie or the shared token. This is the whole security boundary,
so:
request falls through to a session cookie or the shared token. That is the whole security boundary.
- List the **proxy's own address**, not a range. `["127.0.0.1"]` when the tunnel runs beside ipx;
the container's IP when it does not.
- Never list a LAN subnet. Anyone on your network could then send
`Cf-Access-Authenticated-User-Email: admin@…` and be your admin.
- Make sure the origin port is not reachable *around* the proxy by anyone you would not admit
through it. If it is, bind ipx to `127.0.0.1` and let only the proxy reach it.
With the tunnel reaching ipx through the host's port, `192.168.16.1` means **any container on Tower
that connects to `192.168.1.130:8099`**, not only `cloudflared`. Machines on the LAN, and Tower
itself, arrive under their own addresses and cannot set the header; the checks above show both
sides. Never list a LAN address or range: anyone there could then send
`Cf-Access-Authenticated-User-Email: rays@sdf1.net` and be you.
Verify the refusal, don't assume it — set `trusted_proxies = ["10.9.9.9"]` briefly and confirm a
header from your machine gets a `401`:
**What ipx does not do:** it does not verify Cloudflare's signed `Cf-Access-Jwt-Assertion`. It
trusts the hop. Verifying the signature would make the containers on Tower irrelevant to the
boundary, and is the upgrade if that ever matters.
```sh
curl -s -o /dev/null -w '%{http_code}\n' \
-H 'Cf-Access-Authenticated-User-Email: someone@example.com' http://127.0.0.1:8099/api/me
```
**What ipx does not do:** it does not verify Cloudflare's `Cf-Access-Jwt-Assertion` signature or
Authentik's session. It trusts the hop. That is a deliberate trade — it keeps the configuration to
three lines — and it is sound exactly as long as the point above holds.
**Turning it off:** clear `trusted_header`. Existing proxy-only accounts stay, but nobody can sign
**Turning it off:** clear `trusted_header` and restart. Proxy-made accounts stay, but nobody can sign
in with them until they are given a password (`ipx user passwd <name>`).
---
@@ -207,22 +184,20 @@ in with them until they are given a password (`ipx user passwd <name>`).
## Everyday administration
```sh
ipx user list # who exists, and how each one signs in
ipx user list # who exists, how each signs in, and when
echo -n 'secret123' | ipx user add sam # local account, password on stdin
ipx user add sam --no-password # proxy-only account, created ahead of time
ipx user add sam@example.com --no-password # proxy-only account, made ahead of time
ipx user rename sam sam@example.com # give an account the name the proxy sends
echo -n 'newsecret' | ipx user passwd sam # change a password
ipx user rm sam # remove the account
```
Set `auto_create_users = false` once everyone who should have an account has one. After that the
proxy vouching for an unknown name is logged and refused, rather than quietly making an account.
Pre-create people instead with `ipx user add <name> --no-password`, using exactly the name the
header will carry (Cloudflare sends the email address, lower-cased).
In the container, put `docker exec iPodderX` in front, and `docker exec -i iPodderX` for the ones
that read a password.
Scanning intervals, the disk quota, retention, the download folder and a feed's URL are
**admin-only**: the Settings button is hidden for everyone else, and the API refuses the change even
if the request is made by hand. Everyone controls their own keywords, auto-download, explicit
setting and per-scan cap, along with their own read state and which feeds they see.
Set `auto_create_users = false` once everyone who should have an account has one. After that the
proxy vouching for an unknown name is logged and refused. Make people ahead of time instead, with
the exact name the header will carry.
See also [users.md](users.md) for what several people share, [configuration.md](configuration.md)
for every `[web]` key, and [cli.md](cli.md) for the `ipx user` commands.

View File

@@ -8,7 +8,7 @@ fetch, one parse and one file.
| Yours alone | The same for everyone |
|---|---|
| Read, starred, playback position | The feed's URL |
| Read, kept, playback position | The feed's URL |
| Which feeds you see at all | Its download folder |
| Keywords, auto-download, explicit, per-scan cap | When it is scanned |
| | The file on disk |
@@ -34,10 +34,10 @@ is shared.
Deleting a file deletes everyone's copy. A feed with other subscribers labels the button **Delete
for everyone** and names them in the confirmation, and the server has the last word: if anyone else
has starred the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
has kept the item or not played it yet, `DELETE /api/enclosures/{id}` answers `409` with the
reason, and only `?force=true` goes through.
Retention follows the same rule: starred by anyone keeps a file, and it counts as read only once
Retention follows the same rule: an item anyone kept keeps its file, and it counts as read only once
every subscriber has read it.
## Signing in
@@ -73,12 +73,13 @@ re-subscribing does not pull the back catalogue again.
**Popular** and **Directory** sit at the top of the feed list, above your own feeds. Popular, also
shown in the Add feed dialog, lists the ten feeds with the most subscribers on this server, you
included. Directory lists every one of them A to Z. Your own feeds are marked Subscribed.
It shows a title, artwork and a count, never a URL or who reads it. Feeds from an
OPML subscription are left out, since they come with the OPML. So is anything that looks private: a
login configured for the feed, credentials in its URL, or a key such as `auth=` or `token=` in the
query, or a feed from a paid-feed service such as Patreon or Supercast, which put the key in the
path. Those are someone's paid subscriptions, and listing them would let anyone here read what they
pay for.
It shows a title, artwork and a count, never a URL or who reads it. An OPML subscription is listed
as the feeds inside it, one by one, and never the OPML itself, so you can take just the shows you
want. Anything that looks private is left out: a login configured for the feed, credentials in its URL,
or a key such as `auth=` or `token=` in the query, or a feed from a paid-feed service such as
Patreon or Supercast, which put the key in the path, and any feed inside an OPML that looks private
itself. Those are someone's paid subscriptions, and listing them would let anyone here read what
they pay for.
An admin can do the same from **Settings → Manage users…**: add someone (with a password, or none
for someone the proxy signs in), tick or untick Admin, or remove an account. Removing one takes its

View File

@@ -26,9 +26,6 @@ pub struct General {
/// How often to re-check feeds: "every 30m", "every 4h", "90" (minutes), "1d".
/// A feed's own `schedule` overrides this.
pub schedule: String,
/// Superseded by `schedule`. Still read so existing configs keep working.
#[serde(skip_serializing_if = "Option::is_none")]
pub interval_mins: Option<u64>,
pub organize: Organize,
/// 0 = unlimited.
pub max_total_gb: f64,
@@ -83,6 +80,10 @@ pub struct Web {
pub trusted_proxies: Vec<String>,
/// Create an account the first time the proxy vouches for a name it has not seen.
pub auto_create_users: bool,
/// Where Sign out sends someone the proxy signed in. Signing out of ipx alone cannot stick
/// while the proxy still vouches for them, so this is the proxy's own sign-out:
/// `/cdn-cgi/access/logout` behind Cloudflare Access. Empty sends them to /login.
pub sign_out_url: String,
/// Sign a session out after this long without a request.
pub session_days: i64,
}
@@ -96,6 +97,7 @@ impl Default for Web {
trusted_header: String::new(),
trusted_proxies: vec!["127.0.0.1".into(), "::1".into()],
auto_create_users: true,
sign_out_url: String::new(),
session_days: 30,
}
}
@@ -153,7 +155,6 @@ impl Default for General {
download_dir: home().join("Podcasts"),
socket: default_socket(),
schedule: "every 60m".into(),
interval_mins: None,
organize: Organize::Feed,
max_total_gb: 0.0,
max_age_days: 0,
@@ -176,8 +177,8 @@ impl Default for Torrent {
}
impl General {
/// Minutes between checks. Falls back to the legacy `interval_mins`, then to an hour.
/// A malformed value warns rather than stopping the daemon.
/// Minutes between checks, or an hour when `schedule` is empty or unreadable. A malformed
/// value warns rather than stopping the daemon.
pub fn interval(&self) -> u64 {
if let Some(n) = parse_interval(&self.schedule) {
return n;
@@ -185,7 +186,7 @@ impl General {
if !self.schedule.trim().is_empty() {
tracing::warn!(schedule = %self.schedule, "unrecognised schedule; using the default");
}
self.interval_mins.filter(|n| *n > 0).unwrap_or(60)
60
}
}
@@ -278,9 +279,7 @@ pub fn config_path() -> PathBuf {
if let Ok(p) = std::env::var("IPX_CONFIG") {
return PathBuf::from(p);
}
dirs::config_dir()
.unwrap_or_else(|| home().join(".config"))
.join("ipx/config.toml")
xdg("XDG_CONFIG_HOME", ".config").join("ipx/config.toml")
}
/// `$IPX_DATA_DIR`, else `$XDG_DATA_HOME/ipx`.
@@ -288,9 +287,7 @@ pub fn data_dir() -> PathBuf {
if let Ok(p) = std::env::var("IPX_DATA_DIR") {
return PathBuf::from(p);
}
dirs::data_dir()
.unwrap_or_else(|| home().join(".local/share"))
.join("ipx")
xdg("XDG_DATA_HOME", ".local/share").join("ipx")
}
fn default_socket() -> PathBuf {
@@ -346,8 +343,16 @@ pub fn unique_slug(text: &str, taken: &BTreeMap<String, Feed>) -> String {
(2..).map(|n| format!("{base}-{n}")).find(|s| !taken.contains_key(s)).unwrap()
}
/// `$var`, or `~/fallback` when it is unset or empty, as the XDG base directory spec says.
fn xdg(var: &str, fallback: &str) -> PathBuf {
std::env::var_os(var)
.filter(|v| !v.is_empty())
.map(PathBuf::from)
.unwrap_or_else(|| home().join(fallback))
}
fn home() -> PathBuf {
dirs::home_dir().unwrap_or_else(|| PathBuf::from("."))
std::env::var_os("HOME").map(PathBuf::from).unwrap_or_else(|| PathBuf::from("."))
}
fn expand_tilde(p: &Path) -> PathBuf {
@@ -367,6 +372,8 @@ mod tests {
r#"
[general]
download_dir = "/tmp/pods"
# A key older versions read. An old config that still has it has to load.
interval_mins = 45
[feeds.example]
url = "https://example.com/feed.xml"
@@ -403,22 +410,17 @@ mod tests {
}
#[test]
fn interval_falls_back_through_legacy_then_default() {
fn interval_falls_back_to_an_hour() {
let mut g = General::default();
assert_eq!(g.interval(), 60, "the default schedule");
g.schedule = "every 15m".into();
assert_eq!(g.interval(), 15);
// A config written before `schedule` existed still works.
// Empty or garbage must not stop the daemon.
g.schedule = String::new();
g.interval_mins = Some(45);
assert_eq!(g.interval(), 45);
// Garbage must not stop the daemon.
assert_eq!(g.interval(), 60);
g.schedule = "whenever".into();
assert_eq!(g.interval(), 45);
g.interval_mins = None;
assert_eq!(g.interval(), 60);
}

558
src/db.rs
View File

@@ -22,6 +22,10 @@ CREATE TABLE IF NOT EXISTS feeds (
last_checked INTEGER,
ttl_mins INTEGER,
last_error TEXT,
-- When the current run of failures began; NULL while the feed is healthy. Kept
-- through repeated failures so the UI can tell a blip (macmanx: failed once, fine an
-- hour later) from a feed that has been down for a day.
error_since INTEGER,
-- Came from a subscribed OPML that no longer lists it, but has downloads, so kept.
orphaned INTEGER NOT NULL DEFAULT 0,
-- The OPML subscription this feed came from.
@@ -40,14 +44,10 @@ CREATE TABLE IF NOT EXISTS entries (
published INTEGER,
description TEXT,
first_seen INTEGER NOT NULL,
read INTEGER NOT NULL DEFAULT 0,
flagged INTEGER NOT NULL DEFAULT 0,
image TEXT,
duration INTEGER,
episode INTEGER,
season INTEGER,
-- Seconds into the audio, so playback resumes where it was left.
position INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (feed_id, guid)
);
@@ -77,7 +77,9 @@ CREATE TABLE IF NOT EXISTS users (
name TEXT NOT NULL UNIQUE COLLATE NOCASE,
pass_hash TEXT,
is_admin INTEGER NOT NULL DEFAULT 0,
created INTEGER NOT NULL
-- For whoever maintains the server. NULL where it is not known.
created INTEGER,
last_login INTEGER
);
-- What one person wants from a feed. The feed, its items and its files are shared; this
@@ -89,11 +91,10 @@ CREATE TABLE IF NOT EXISTS subscriptions (
auto_download INTEGER,
allow_explicit INTEGER,
max_new_per_check INTEGER,
created INTEGER NOT NULL,
PRIMARY KEY (user_id, feed_id)
);
-- Read, starred and how far in. One row per person per item, created on first touch;
-- Read, kept and how far in. One row per person per item, created on first touch;
-- an item nobody has touched has no row at all, which is what unread means.
CREATE TABLE IF NOT EXISTS entry_state (
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
@@ -108,7 +109,6 @@ CREATE TABLE IF NOT EXISTS entry_state (
CREATE TABLE IF NOT EXISTS sessions (
token TEXT PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
created INTEGER NOT NULL,
seen INTEGER NOT NULL
);
";
@@ -131,6 +131,23 @@ pub struct User {
pub name: String,
pub pass_hash: Option<String>,
pub is_admin: bool,
/// When the account was made and when it last signed in, for whoever maintains the server.
pub created: Option<i64>,
pub last_login: Option<i64>,
}
/// The columns `user_row` reads, in its order.
const USER_COLS: &str = "id, name, pass_hash, is_admin, created, last_login";
fn user_row(r: &rusqlite::Row<'_>) -> rusqlite::Result<User> {
Ok(User {
id: r.get(0)?,
name: r.get(1)?,
pass_hash: r.get(2)?,
is_admin: r.get::<_, i64>(3)? != 0,
created: r.get(4)?,
last_login: r.get(5)?,
})
}
/// A feed derived from an OPML subscription rather than written into the config.
@@ -140,33 +157,49 @@ pub struct Managed {
pub url: String,
pub title: Option<String>,
pub group_id: String,
pub orphaned: bool,
}
/// Adds columns that later versions introduced. CREATE TABLE IF NOT EXISTS does nothing to
/// a table that already exists, so an installed database needs them added explicitly.
/// Adds the columns later versions introduced and drops the ones they retired. CREATE TABLE IF
/// NOT EXISTS leaves a table that already exists alone, so an installed database needs both done
/// explicitly. Columns from before 0.3.0, the oldest version an upgrade may start from, need no
/// entry.
fn migrate(conn: &Connection) -> Result<()> {
let wanted: &[(&str, &str, &str)] = &[
("feeds", "image", "TEXT"),
("feeds", "orphaned", "INTEGER NOT NULL DEFAULT 0"),
("feeds", "group_id", "TEXT"),
("feeds", "managed", "INTEGER NOT NULL DEFAULT 0"),
("entries", "image", "TEXT"),
("entries", "duration", "INTEGER"),
("entries", "episode", "INTEGER"),
("entries", "season", "INTEGER"),
("entries", "position", "INTEGER NOT NULL DEFAULT 0"),
// For whoever maintains the server. An audit dropped `created` as unread on 2026-09-12,
// and it came back the same day with `last_login` beside it.
("users", "created", "INTEGER"),
("users", "last_login", "INTEGER"),
("feeds", "error_since", "INTEGER"),
];
let retired: &[(&str, &str)] = &[
// Read state from before accounts, long since moved to entry_state. Two bugs came from
// queries still reading these after they stopped meaning anything.
("entries", "read"),
("entries", "flagged"),
("entries", "position"),
// Written by every insert and read by nothing.
("subscriptions", "created"),
("sessions", "created"),
];
let has = |table: &str, column: &str| -> Result<bool> {
let names: Vec<String> = conn
.prepare(&format!("PRAGMA table_info({table})"))?
.query_map([], |r| r.get(1))?
.collect::<rusqlite::Result<_>>()?;
Ok(names.iter().any(|c| c == column))
};
for (table, column, ty) in wanted {
let mut stmt = conn.prepare(&format!("PRAGMA table_info({table})"))?;
let existing: Vec<String> = stmt
.query_map([], |r| r.get::<_, String>(1))?
.collect::<rusqlite::Result<Vec<_>>>()?;
if !existing.iter().any(|c| c == column) {
if !has(table, column)? {
tracing::info!(table, column, "adding column");
conn.execute_batch(&format!("ALTER TABLE {table} ADD COLUMN {column} {ty}"))?;
}
}
for (table, column) in retired {
if has(table, column)? {
tracing::info!(table, column, "dropping column");
conn.execute_batch(&format!("ALTER TABLE {table} DROP COLUMN {column}"))?;
}
}
Ok(())
}
@@ -180,6 +213,8 @@ pub struct FeedSummary {
pub orphaned: bool,
pub last_checked: Option<i64>,
pub last_error: Option<String>,
/// When this run of failures began; see the `error_since` column.
pub error_since: Option<i64>,
pub entries: i64,
pub downloaded: i64,
}
@@ -221,7 +256,7 @@ impl Db {
let conn = self.conn.lock().unwrap();
let mut sum: FeedSummary = conn
.query_row(
"SELECT title, image, last_checked, last_error, coalesce(orphaned, 0)
"SELECT title, image, last_checked, last_error, coalesce(orphaned, 0), error_since
FROM feeds WHERE id = ?1",
[feed_id],
|r| {
@@ -231,6 +266,7 @@ impl Db {
last_checked: r.get(2)?,
last_error: r.get(3)?,
orphaned: r.get::<_, i64>(4)? != 0,
error_since: r.get(5)?,
..Default::default()
})
},
@@ -305,7 +341,8 @@ impl Db {
last_checked = excluded.last_checked,
ttl_mins = excluded.ttl_mins,
image = coalesce(excluded.image, feeds.image),
last_error = NULL",
last_error = NULL,
error_since = NULL",
rusqlite::params![feed_id, url, title, etag, last_modified, now(), ttl_mins.map(|t| t as i64), image],
)?;
Ok(())
@@ -316,7 +353,8 @@ impl Db {
let conn = self.conn.lock().unwrap();
conn.execute(
"INSERT INTO feeds (id, url, last_checked) VALUES (?1, ?2, ?3)
ON CONFLICT(id) DO UPDATE SET last_checked = excluded.last_checked, last_error = NULL",
ON CONFLICT(id) DO UPDATE SET last_checked = excluded.last_checked,
last_error = NULL, error_since = NULL",
rusqlite::params![feed_id, url, now()],
)?;
Ok(())
@@ -335,32 +373,34 @@ impl Db {
pub fn set_feed_error(&self, feed_id: &str, url: &str, msg: &str) -> Result<()> {
let conn = self.conn.lock().unwrap();
let now = now();
conn.execute(
"INSERT INTO feeds (id, url, last_checked, last_error) VALUES (?1, ?2, ?3, ?4)
ON CONFLICT(id) DO UPDATE SET last_checked = excluded.last_checked, last_error = excluded.last_error",
rusqlite::params![feed_id, url, now(), msg],
"INSERT INTO feeds (id, url, last_checked, last_error, error_since)
VALUES (?1, ?2, ?3, ?4, ?3)
ON CONFLICT(id) DO UPDATE SET
last_checked = excluded.last_checked,
last_error = excluded.last_error,
error_since = coalesce(feeds.error_since, excluded.error_since)",
rusqlite::params![feed_id, url, now, msg],
)?;
Ok(())
}
/// Returns true when this entry had not been seen before.
///
/// A changed description or title flips `read` back to 0, which is what the original's
/// textDiff dance was ultimately for -- minus the diff markup, which the UI can do.
pub fn record_entry(&self, feed_id: &str, e: &crate::feed::Entry) -> Result<bool> {
let conn = self.conn.lock().unwrap();
let inserted = conn.execute(
"INSERT OR IGNORE INTO entries
(feed_id, guid, title, link, published, description, first_seen, read, flagged,
(feed_id, guid, title, link, published, description, first_seen,
image, duration, episode, season)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, 0, 0, ?8, ?9, ?10, ?11)",
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)",
rusqlite::params![
feed_id, e.guid, e.title, e.link, e.published, e.description, now(),
e.image, e.duration, e.episode, e.season
],
)?;
if inserted == 0 {
// The SET expressions see the pre-update row, so this compares old vs new.
conn.execute(
"UPDATE entries SET
title = coalesce(?3, title),
@@ -368,8 +408,7 @@ impl Db {
image = coalesce(?5, image),
duration = coalesce(?6, duration),
episode = coalesce(?7, episode),
season = coalesce(?8, season),
read = CASE WHEN description IS NOT ?4 OR title IS NOT ?3 THEN 0 ELSE read END
season = coalesce(?8, season)
WHERE feed_id = ?1 AND guid = ?2",
rusqlite::params![
feed_id, e.guid, e.title, e.description,
@@ -687,22 +726,9 @@ pub struct EncRow {
}
impl Db {
/// One page of a feed's entries, newest first, each with its enclosures attached.
/// `search` matches title and description, case-insensitively.
pub fn entries(
&self,
user_id: i64,
feed_id: &str,
filter: Filter,
search: Option<&str>,
offset: i64,
limit: i64,
) -> Result<Vec<EntryRow>> {
self.entries_in(user_id, Some(feed_id), filter, search, offset, limit, &order_sql("published", "desc"))
}
/// `entries` for one feed, or across every feed the person subscribes to when `feed_id`
/// is None: the All Subscriptions view.
/// One page of entries, each with its enclosures attached: one feed's, or every feed the
/// person subscribes to when `feed_id` is None (All Subscriptions). `search` matches title
/// and description, case-insensitively.
pub fn entries_in(
&self,
user_id: i64,
@@ -793,18 +819,7 @@ impl Db {
Ok(rows)
}
/// How many entries match, so the UI knows whether there is another page.
pub fn count_entries(
&self,
user_id: i64,
feed_id: &str,
filter: Filter,
search: Option<&str>,
) -> Result<i64> {
self.count_in(user_id, Some(feed_id), filter, search)
}
/// `count_entries` for one feed, or across every feed the person subscribes to.
/// How many entries `entries_in` would page through, so the UI knows whether there is more.
pub fn count_in(
&self,
user_id: i64,
@@ -838,37 +853,28 @@ impl Db {
Ok(())
}
/// Marks every entry in a feed read, for the "mark all read" button.
/// Moves a single-user library onto an account: everything read, starred or part-played
/// becomes that person's, and they subscribe to every feed already in the catalogue.
/// Runs once -- the moment there is a first account and no subscriptions yet.
pub fn adopt_existing_library(&self, user_id: i64, catalogue: &[String]) -> Result<usize> {
/// The first admin starts subscribed to the whole catalogue: whoever wrote config.toml meant
/// to read those feeds, and without this a fresh install signs in to an empty sidebar. Runs
/// only while nobody subscribes to anything, so an unsubscribe is never undone.
pub fn adopt_catalogue(&self, user_id: i64, catalogue: &[String]) -> Result<usize> {
let conn = self.conn.lock().unwrap();
let already: i64 =
conn.query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0))?;
if already > 0 {
return Ok(0);
}
let moved = conn.execute(
"INSERT INTO entry_state (user_id, feed_id, guid, read, flagged, position)
SELECT ?1, feed_id, guid, read, flagged, position FROM entries
WHERE read = 1 OR flagged = 1 OR position > 0
ON CONFLICT(user_id, feed_id, guid) DO NOTHING",
[user_id],
)?;
for id in catalogue {
conn.execute(
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created) VALUES (?1, ?2, ?3)",
params![user_id, id, now()],
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id) VALUES (?1, ?2)",
params![user_id, id],
)?;
}
// Feeds that exist only in the database (OPML children) count too.
conn.execute(
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created)
SELECT ?1, id, ?2 FROM feeds",
params![user_id, now()],
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id) SELECT ?1, id FROM feeds",
params![user_id],
)?;
Ok(moved)
Ok(catalogue.len())
}
// ---- subscriptions ----
@@ -920,14 +926,20 @@ impl Db {
/// Everyone's settings for one feed. The scanner merges these into what it fetches
/// and downloads, since one file serves the lot.
pub fn subscribers(&self, feed_id: &str) -> Result<Vec<Sub>> {
/// In a group, whatever someone has not set on the feed itself comes from their
/// subscription to the group, as the group's settings dialog has always said it does.
pub fn subscribers(&self, feed_id: &str, group: Option<&str>) -> Result<Vec<Sub>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare(
"SELECT keywords, auto_download, allow_explicit, max_new_per_check
FROM subscriptions WHERE feed_id = ?1",
"SELECT coalesce(c.keywords, p.keywords), coalesce(c.auto_download, p.auto_download),
coalesce(c.allow_explicit, p.allow_explicit),
coalesce(c.max_new_per_check, p.max_new_per_check)
FROM subscriptions c
LEFT JOIN subscriptions p ON p.user_id = c.user_id AND p.feed_id = ?2
WHERE c.feed_id = ?1",
)?;
let out = stmt
.query_map([feed_id], |r| {
.query_map(params![feed_id, group], |r| {
Ok(Sub {
feed_id: feed_id.to_string(),
keywords: r
@@ -977,8 +989,8 @@ impl Db {
pub fn subscribe(&self, user_id: i64, feed_id: &str) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute(
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id, created) VALUES (?1, ?2, ?3)",
params![user_id, feed_id, now()],
"INSERT OR IGNORE INTO subscriptions (user_id, feed_id) VALUES (?1, ?2)",
params![user_id, feed_id],
)?;
Ok(())
}
@@ -992,16 +1004,6 @@ impl Db {
Ok(())
}
/// How many people want this feed. Nobody means it stops being scanned.
pub fn subscriber_count(&self, feed_id: &str) -> Result<i64> {
let conn = self.conn.lock().unwrap();
Ok(conn.query_row(
"SELECT count(*) FROM subscriptions WHERE feed_id = ?1",
[feed_id],
|r| r.get(0),
)?)
}
/// Overwrites one person's settings for a feed. A None field means: follow the feed.
pub fn set_subscription(&self, user_id: i64, sub: &Sub) -> Result<()> {
let conn = self.conn.lock().unwrap();
@@ -1012,8 +1014,8 @@ impl Db {
.transpose()?;
conn.execute(
"INSERT INTO subscriptions
(user_id, feed_id, keywords, auto_download, allow_explicit, max_new_per_check, created)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
(user_id, feed_id, keywords, auto_download, allow_explicit, max_new_per_check)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
ON CONFLICT(user_id, feed_id) DO UPDATE SET
keywords = excluded.keywords,
auto_download = excluded.auto_download,
@@ -1026,22 +1028,11 @@ impl Db {
sub.auto_download.map(|v| v as i64),
sub.allow_explicit.map(|v| v as i64),
sub.max_new_per_check,
now()
],
)?;
Ok(())
}
/// Feeds with at least one subscriber. What the scanner walks.
pub fn subscribed_feed_ids(&self) -> Result<Vec<String>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare("SELECT DISTINCT feed_id FROM subscriptions")?;
let out = stmt
.query_map([], |r| r.get::<_, String>(0))?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(out)
}
// ---- users and sessions ----
pub fn create_user(&self, name: &str, pass_hash: Option<&str>, admin: bool) -> Result<i64> {
@@ -1054,11 +1045,11 @@ impl Db {
}
pub fn user_by_name(&self, name: &str) -> Result<Option<User>> {
self.one_user("SELECT id, name, pass_hash, is_admin FROM users WHERE name = ?1", name)
self.one_user(&format!("SELECT {USER_COLS} FROM users WHERE name = ?1"), name)
}
pub fn user_by_id(&self, id: i64) -> Result<Option<User>> {
self.one_user("SELECT id, name, pass_hash, is_admin FROM users WHERE id = ?1", id)
self.one_user(&format!("SELECT {USER_COLS} FROM users WHERE id = ?1"), id)
}
fn one_user<P: rusqlite::ToSql>(&self, sql: &str, key: P) -> Result<Option<User>> {
@@ -1066,12 +1057,7 @@ impl Db {
let mut stmt = conn.prepare(sql)?;
let mut rows = stmt.query(params![key])?;
Ok(match rows.next()? {
Some(r) => Some(User {
id: r.get(0)?,
name: r.get(1)?,
pass_hash: r.get(2)?,
is_admin: r.get::<_, i64>(3)? != 0,
}),
Some(r) => Some(user_row(r)?),
None => None,
})
}
@@ -1079,16 +1065,9 @@ impl Db {
pub fn users(&self) -> Result<Vec<User>> {
let conn = self.conn.lock().unwrap();
let mut stmt =
conn.prepare("SELECT id, name, pass_hash, is_admin FROM users ORDER BY name")?;
conn.prepare(&format!("SELECT {USER_COLS} FROM users ORDER BY name"))?;
let out = stmt
.query_map([], |r| {
Ok(User {
id: r.get(0)?,
name: r.get(1)?,
pass_hash: r.get(2)?,
is_admin: r.get::<_, i64>(3)? != 0,
})
})?
.query_map([], user_row)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(out)
}
@@ -1105,6 +1084,26 @@ impl Db {
Ok(())
}
/// The proxy signs people in by the name it vouches for, so an account made before the proxy
/// was set up has to take that name to be found by it. The name is UNIQUE, so a taken one is
/// refused here as well as by the caller.
pub fn rename_user(&self, id: i64, name: &str) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute("UPDATE users SET name = ?2 WHERE id = ?1", params![id, name])?;
Ok(())
}
/// Records a sign-in, to the hour: the proxy vouches for every request, and writing each one
/// would buy nothing.
pub fn signed_in(&self, id: i64) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute(
"UPDATE users SET last_login = ?2 WHERE id = ?1 AND coalesce(last_login, 0) <= ?2 - 3600",
params![id, now()],
)?;
Ok(())
}
/// Sessions go with the user: a deleted account must not leave a usable cookie behind.
pub fn delete_user(&self, id: i64) -> Result<()> {
let conn = self.conn.lock().unwrap();
@@ -1116,7 +1115,7 @@ impl Db {
pub fn create_session(&self, user_id: i64, token: &str) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute(
"INSERT INTO sessions (token, user_id, created, seen) VALUES (?1, ?2, ?3, ?3)",
"INSERT INTO sessions (token, user_id, seen) VALUES (?1, ?2, ?3)",
params![token, user_id, now()],
)?;
Ok(())
@@ -1128,18 +1127,13 @@ impl Db {
let conn = self.conn.lock().unwrap();
let cutoff = now() - max_idle_secs;
let mut stmt = conn.prepare(
"SELECT u.id, u.name, u.pass_hash, u.is_admin
"SELECT u.id, u.name, u.pass_hash, u.is_admin, u.created, u.last_login
FROM sessions s JOIN users u ON u.id = s.user_id
WHERE s.token = ?1 AND s.seen >= ?2",
)?;
let mut rows = stmt.query(params![token, cutoff])?;
let found = match rows.next()? {
Some(r) => Some(User {
id: r.get(0)?,
name: r.get(1)?,
pass_hash: r.get(2)?,
is_admin: r.get::<_, i64>(3)? != 0,
}),
Some(r) => Some(user_row(r)?),
None => None,
};
drop(rows);
@@ -1218,7 +1212,7 @@ impl Db {
.optional()?)
}
/// Read and starred, per person. The row is created on first touch.
/// Read and kept, per person. The row is created on first touch.
pub fn set_entry_flag(
&self,
user_id: i64,
@@ -1293,7 +1287,7 @@ impl Db {
pub fn managed_feeds(&self) -> Result<Vec<Managed>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare(
"SELECT id, url, title, group_id, orphaned FROM feeds
"SELECT id, url, title, group_id FROM feeds
WHERE managed = 1 AND group_id IS NOT NULL ORDER BY coalesce(title, id)",
)?;
Ok(stmt
@@ -1303,7 +1297,6 @@ impl Db {
url: r.get(1)?,
title: r.get(2)?,
group_id: r.get(3)?,
orphaned: r.get::<_, i64>(4)? != 0,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?)
@@ -1325,6 +1318,71 @@ impl Db {
Ok(())
}
/// Empties a feed of its items, leaving its files alone.
pub fn clear_entries(&self, feed_id: &str) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute("DELETE FROM entries WHERE feed_id = ?1", [feed_id])?;
Ok(())
}
/// Every feed the database holds rows for, as (id, url), removed ones included.
pub fn feed_urls(&self) -> Result<Vec<(String, String)>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare("SELECT id, coalesce(url, '') FROM feeds")?;
let out = stmt
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))?
.collect::<rusqlite::Result<_>>()?;
Ok(out)
}
/// Hands a feed in a group the enclosures its parent holds, as (guid, url), with everyone's
/// read state for them. A Patreon creator read as one feed before it was split into shows
/// owns every show's files, and `enclosures.url` is unique, so without this each show would
/// list its items with nothing to play.
pub fn adopt(&self, parent: &str, child: &str, listed: &[(&str, &str)]) -> Result<()> {
let mut conn = self.conn.lock().unwrap();
let holds: bool = conn.query_row(
"SELECT EXISTS (SELECT 1 FROM enclosures WHERE feed_id = ?1)",
[parent],
|r| r.get(0),
)?;
if !holds {
return Ok(()); // An OPML, or a creator already shared out.
}
let tx = conn.transaction()?;
for &(guid, url) in listed {
let moved = tx.execute(
"UPDATE enclosures SET feed_id = ?3, guid = ?4 WHERE url = ?1 AND feed_id = ?2",
params![url, parent, child, guid],
)?;
if moved == 1 {
// Patreon gives a post the same guid in every feed it appears in.
tx.execute(
"UPDATE OR IGNORE entry_state SET feed_id = ?2 WHERE feed_id = ?1 AND guid = ?3",
params![parent, child, guid],
)?;
}
}
tx.commit()?;
Ok(())
}
/// A feed's enclosures skipped by one of its filters, by URL, with the reason: the verdicts a
/// change of settings can overturn. A torrent held back while torrents are off is not a
/// filter's call.
pub fn skipped_by_filter(&self, feed_id: &str) -> Result<std::collections::HashMap<String, String>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare(
"SELECT url, last_error FROM enclosures
WHERE feed_id = ?1 AND state = 'skipped' AND last_error IS NOT NULL
AND last_error != 'torrents disabled'",
)?;
let out = stmt
.query_map([feed_id], |r| Ok((r.get(0)?, r.get(1)?)))?
.collect::<rusqlite::Result<_>>()?;
Ok(out)
}
pub fn set_orphaned(&self, feed_id: &str, on: bool) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute(
@@ -1381,8 +1439,8 @@ mod tests {
fn every_sort_column_runs_and_orders_both_ways() {
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0);
INSERT INTO subscriptions (user_id, feed_id, created) VALUES (1,'f',0),(1,'g',0);
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(1,'g');
INSERT INTO feeds (id, url, title) VALUES ('f','u','Zebra'),('g','v','Aardvark');
INSERT INTO entries (feed_id, guid, title, first_seen) VALUES
('f','a','banana',100),('g','b','Apple',200),('f','c','cherry',300);
@@ -1415,8 +1473,8 @@ mod tests {
fn deleting_a_shared_file_asks_about_everyone_else() {
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0),(3,'kit',0,0);
INSERT INTO subscriptions (user_id, feed_id, created) VALUES (1,'f',0),(2,'f',0),(3,'f',0);
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0),(3,'kit',0);
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f'),(3,'f');
INSERT INTO entries (feed_id, guid, first_seen) VALUES ('f','a',0);
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
(1,'f','a','u1','/tmp/a','done');",
@@ -1441,7 +1499,7 @@ mod tests {
fn read_state_belongs_to_one_person() {
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0);
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
INSERT INTO entries (feed_id, guid, title, first_seen) VALUES
('f','a','One',100),('f','b','Two',200);",
)
@@ -1457,8 +1515,9 @@ mod tests {
// Starring and position are just as private.
db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).unwrap();
db.set_position(2, "f", "b", 42).unwrap();
let ray = db.entries(1, "f", Filter::All, None, 0, 50).unwrap();
let sam = db.entries(2, "f", Filter::All, None, 0, 50).unwrap();
let order = order_sql("published", "desc");
let page = |user| db.entries_in(user, Some("f"), Filter::All, None, 0, 50, &order).unwrap();
let (ray, sam) = (page(1), page(2));
let ray_b = ray.iter().find(|e| e.guid == "b").unwrap();
let sam_b = sam.iter().find(|e| e.guid == "b").unwrap();
assert!(ray_b.flagged && ray_b.position == 0);
@@ -1482,6 +1541,111 @@ mod tests {
assert_eq!(sum.downloaded, 0);
}
#[test]
fn an_old_database_loses_its_retired_columns() {
let conn = Connection::open_in_memory().unwrap();
// As open() has it: a DROP COLUMN on a table that references another is the part worth
// proving, and it has to work with the foreign keys switched on.
conn.pragma_update(None, "foreign_keys", "ON").unwrap();
conn.execute_batch(
"CREATE TABLE entries (feed_id TEXT NOT NULL, guid TEXT NOT NULL,
first_seen INTEGER NOT NULL, read INTEGER NOT NULL DEFAULT 0,
flagged INTEGER NOT NULL DEFAULT 0, position INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (feed_id, guid));
CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL UNIQUE COLLATE NOCASE,
pass_hash TEXT, is_admin INTEGER NOT NULL DEFAULT 0, created INTEGER NOT NULL);
CREATE TABLE subscriptions (
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
feed_id TEXT NOT NULL, created INTEGER NOT NULL, PRIMARY KEY (user_id, feed_id));
CREATE TABLE sessions (token TEXT PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
created INTEGER NOT NULL, seen INTEGER NOT NULL);
INSERT INTO users VALUES (1, 'ray', NULL, 1, 0);
INSERT INTO subscriptions VALUES (1, 'f', 0);
INSERT INTO sessions VALUES ('t', 1, 0, 0);",
)
.unwrap();
// The same order as open(): the schema leaves the old tables alone, migrate() fixes them.
conn.execute_batch(SCHEMA).unwrap();
migrate(&conn).unwrap();
let cols = |table: &str| -> Vec<String> {
conn.prepare(&format!("PRAGMA table_info({table})"))
.unwrap()
.query_map([], |r| r.get(1))
.unwrap()
.collect::<rusqlite::Result<_>>()
.unwrap()
};
for (table, gone) in [
("entries", &["read", "flagged", "position"][..]),
("subscriptions", &["created"][..]),
("sessions", &["created"][..]),
] {
let cols = cols(table);
assert!(!cols.iter().any(|c| gone.contains(&c.as_str())), "{table}: {cols:?}");
}
// users.created is not retired: it keeps what it held, and last_login joins it.
let users = cols("users");
assert!(users.iter().any(|c| c == "last_login"), "{users:?}");
assert_eq!(conn.query_row("SELECT created FROM users", [], |r| r.get::<_, i64>(0)).unwrap(), 0);
// And the rows come through it.
let kept: i64 = conn
.query_row("SELECT count(*) FROM subscriptions JOIN sessions USING (user_id)", [], |r| r.get(0))
.unwrap();
assert_eq!(kept, 1);
}
#[test]
fn a_renamed_account_keeps_everything_but_its_name() {
let db = Db::memory().unwrap();
let ray = db.create_user("rays", None, true).unwrap();
db.create_user("sam", None, false).unwrap();
db.subscribe(ray, "f").unwrap();
db.rename_user(ray, "rays@sdf1.net").unwrap();
assert!(db.user_by_name("rays").unwrap().is_none());
let renamed = db.user_by_name("RAYS@sdf1.net").unwrap().unwrap();
assert_eq!((renamed.id, renamed.is_admin), (ray, true), "same account, still the admin");
assert_eq!(db.subscriptions_for(ray).unwrap().len(), 1, "and still subscribed");
assert!(db.rename_user(ray, "sam").is_err(), "a taken name is refused");
}
#[test]
fn an_account_knows_when_it_was_made_and_last_signed_in() {
let db = Db::memory().unwrap();
let id = db.create_user("ray", None, true).unwrap();
let get = || db.user_by_id(id).unwrap().unwrap();
assert!(get().created.is_some_and(|t| t > 0));
assert_eq!(get().last_login, None, "made, but never signed in");
db.signed_in(id).unwrap();
let first = get().last_login.unwrap();
// Within the hour, the proxy vouching again writes nothing; after it, it does.
db.exec_for_test(&format!("UPDATE users SET last_login = {} WHERE id = {id}", first - 60)).unwrap();
db.signed_in(id).unwrap();
assert_eq!(get().last_login, Some(first - 60));
db.exec_for_test(&format!("UPDATE users SET last_login = {} WHERE id = {id}", first - 7200)).unwrap();
db.signed_in(id).unwrap();
assert!(get().last_login.unwrap() >= first);
}
#[test]
fn the_first_admin_starts_with_the_catalogue_and_only_once() {
// Cutting this along with the dead read columns left the browser suite's admin with an
// empty sidebar: it is how a fresh install's first account gets config.toml's feeds.
let db = Db::memory().unwrap();
db.exec_for_test("INSERT INTO users (id, name, is_admin) VALUES (1,'admin',1);")
.unwrap();
let subs = || -> i64 {
db.conn.lock().unwrap().query_row("SELECT count(*) FROM subscriptions", [], |r| r.get(0)).unwrap()
};
let catalogue = ["a".to_string(), "b".to_string()];
assert_eq!(db.adopt_catalogue(1, &catalogue).unwrap(), 2);
assert_eq!(subs(), 2);
// Once anyone subscribes to anything it never runs again, so an unsubscribe sticks.
db.unsubscribe(1, "a").unwrap();
assert_eq!(db.adopt_catalogue(1, &catalogue).unwrap(), 0);
assert_eq!(subs(), 1);
}
#[test]
fn every_filter_works_with_and_without_a_search_term() {
// Regression: the search clause used to be omitted when no term was given, while
@@ -1496,7 +1660,7 @@ mod tests {
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
(1,'f','b','u1','/tmp/b','done');
-- Read and starred belong to a person now, so say which one.
INSERT INTO users (id, name, is_admin, created) VALUES (7,'reader',1,0);
INSERT INTO users (id, name, is_admin) VALUES (7,'reader',1);
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
(7,'f','b',1,0),
(7,'f','c',1,1);",
@@ -1505,21 +1669,22 @@ mod tests {
for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged] {
// Both paths must run without erroring, and agree with each other.
let rows = db.entries(7, "f", f, None, 0, 50).unwrap();
let n = db.count_entries(7, "f", f, None).unwrap();
let order = order_sql("published", "desc");
let rows = db.entries_in(7, Some("f"), f, None, 0, 50, &order).unwrap();
let n = db.count_in(7, Some("f"), f, None).unwrap();
assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page");
let rows = db.entries(7, "f", f, Some("dive"), 0, 50).unwrap();
let n = db.count_entries(7, "f", f, Some("dive")).unwrap();
let rows = db.entries_in(7, Some("f"), f, Some("dive"), 0, 50, &order).unwrap();
let n = db.count_in(7, Some("f"), f, Some("dive")).unwrap();
assert_eq!(rows.len() as i64, n, "{f:?} with search disagrees");
}
assert_eq!(db.count_entries(7, "f", Filter::All, None).unwrap(), 3);
assert_eq!(db.count_entries(7, "f", Filter::Unread, None).unwrap(), 1);
assert_eq!(db.count_entries(7, "f", Filter::Downloaded, None).unwrap(), 1);
assert_eq!(db.count_entries(7, "f", Filter::Flagged, None).unwrap(), 1);
assert_eq!(db.count_entries(7, "f", Filter::All, Some("dive")).unwrap(), 2);
assert_eq!(db.count_entries(7, "f", Filter::All, Some("NOTES two")).unwrap(), 1,
assert_eq!(db.count_in(7, Some("f"), Filter::All, None).unwrap(), 3);
assert_eq!(db.count_in(7, Some("f"), Filter::Unread, None).unwrap(), 1);
assert_eq!(db.count_in(7, Some("f"), Filter::Downloaded, None).unwrap(), 1);
assert_eq!(db.count_in(7, Some("f"), Filter::Flagged, None).unwrap(), 1);
assert_eq!(db.count_in(7, Some("f"), Filter::All, Some("dive")).unwrap(), 2);
assert_eq!(db.count_in(7, Some("f"), Filter::All, Some("NOTES two")).unwrap(), 1,
"search is case-insensitive and covers the description");
}
@@ -1561,6 +1726,81 @@ mod tests {
assert_eq!(state(4), "done");
}
#[test]
fn a_show_takes_over_what_its_creator_held() {
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
INSERT INTO enclosures (id, feed_id, guid, url, state, path, last_error) VALUES
(1,'creator','a','u1','done','/x/a.mp3',NULL),
(2,'creator','b','u2','skipped',NULL,'explicit'),
(3,'creator','c','u3','skipped',NULL,'explicit'),
(4,'other','d','u4','skipped',NULL,'torrents disabled');
INSERT INTO entry_state (user_id, feed_id, guid, read) VALUES (1,'creator','a',1);",
)
.unwrap();
db.adopt("creator", "show", &[("a", "u1"), ("b", "u2"), ("d", "u4")]).unwrap();
{
let conn = db.conn.lock().unwrap();
let owner = |id: i64| -> String {
conn.query_row("SELECT feed_id FROM enclosures WHERE id = ?1", [id], |r| r.get(0)).unwrap()
};
assert_eq!(owner(1), "show", "a downloaded file moves with its item");
assert_eq!(owner(2), "show");
assert_eq!(owner(3), "creator", "this show does not list it");
assert_eq!(owner(4), "other", "only the parent's are taken");
let read: String = conn
.query_row("SELECT feed_id FROM entry_state WHERE user_id = 1 AND guid = 'a'", [], |r| r.get(0))
.unwrap();
assert_eq!(read, "show", "what you had read stays read");
}
// Only a filter's verdict can be overturned by a change of settings.
let skipped = db.skipped_by_filter("show").unwrap();
assert_eq!(skipped.get("u2").map(String::as_str), Some("explicit"));
assert_eq!(skipped.len(), 1);
assert!(db.skipped_by_filter("other").unwrap().is_empty(), "torrents disabled is not a filter");
}
#[test]
fn a_feed_in_a_group_follows_your_settings_on_the_group() {
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
INSERT INTO subscriptions (user_id, feed_id, allow_explicit) VALUES
(1,'group',1),(1,'show',NULL),(2,'group',1),(2,'show',0);",
)
.unwrap();
let explicit = |group| -> Vec<Option<bool>> {
let mut v: Vec<_> =
db.subscribers("show", group).unwrap().into_iter().map(|s| s.allow_explicit).collect();
v.sort();
v
};
assert_eq!(explicit(Some("group")), [Some(false), Some(true)], "ray inherits; sam's own choice on the show wins");
assert_eq!(explicit(None), [None, Some(false)], "outside a group nothing is inherited");
}
#[test]
fn error_since_marks_the_start_of_a_run_of_failures_and_clears_on_success() {
let db = Db::memory().unwrap();
db.set_feed_error("f", "http://x", "HTTP 404").unwrap();
// Backdate it, as if this feed had already been failing a while, so a second
// failure landing "now" is distinguishable from the first.
db.exec_for_test("UPDATE feeds SET error_since = error_since - 3600 WHERE id = 'f'").unwrap();
let first = db.feed_summary("f").unwrap().error_since.unwrap();
// macmanx: failed once, read fine an hour later. A second failure must not push
// error_since forward -- the UI decides "failing for a day" from the first one.
db.set_feed_error("f", "http://x", "HTTP 404").unwrap();
assert_eq!(db.feed_summary("f").unwrap().error_since, Some(first));
db.touch_feed("f", "http://x").unwrap();
let after = db.feed_summary("f").unwrap();
assert_eq!(after.last_error, None);
assert_eq!(after.error_since, None, "a clean check ends the run of failures");
}
#[test]
fn enclosure_url_is_the_dedupe_key() {
let db = Db::memory().unwrap();

View File

@@ -223,7 +223,7 @@ enum Sniffed {
/// 2008 and so always answered 'data'.
async fn sniff(path: &Path) -> Result<Sniffed> {
let head = read_head(path, 512).await?;
if infer::is(&head, "torrent") || head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
if head.starts_with(b"d8:announce") || head.starts_with(b"d7:") {
return Ok(Sniffed::Torrent);
}
let text = String::from_utf8_lossy(&head);

View File

@@ -87,6 +87,50 @@ pub async fn fetch(
Ok(Fetched::Body { bytes, etag, last_modified })
}
/// A stored `last_error`, translated into plain words for whoever subscribes: whose problem
/// it is, and whether there is a new address to switch to.
pub struct Failure {
pub reason: &'static str,
pub new_url: Option<String>,
}
/// Reads a `last_error` the same way `set_feed_error` received it (`format!("{e:#}")` on the
/// anyhow chain from `fetch` or `parse`) and says what it means, for the errors worth telling
/// someone about. Everything else -- a timeout, a 5xx, a 429, a feed that is simply garbled --
/// comes back `None`: transient by nature, or with nothing more useful to say than the raw
/// text already shown once a feed is open.
///
/// ponytail: matches on the fixed strings this crate itself produces (`anyhow!("HTTP
/// {status}")`, and the "got a web page" message above) plus the substrings a DNS failure
/// reliably contains. Fragile if reqwest's own wording changes; the fallback is just showing
/// nothing extra, so a miss costs a clearer message, not a wrong one.
pub fn explain_failure(msg: &str) -> Option<Failure> {
if let Some(rest) = msg.strip_prefix("got a web page, not a feed") {
let new_url = rest
.strip_prefix("; it links ")
.and_then(|r| r.strip_suffix(" as its feed"))
.map(str::to_owned);
return Some(Failure { reason: "The feed moved; this address now shows a web page.", new_url });
}
let low = msg.to_ascii_lowercase();
if low.contains("http 404") {
return Some(Failure { reason: "The publisher took this feed down, or moved it.", new_url: None });
}
if low.contains("http 401") || low.contains("http 403") {
return Some(Failure { reason: "The site refuses ipx's requests.", new_url: None });
}
if low.contains("http 402") {
return Some(Failure { reason: "The feed now needs a paid plan.", new_url: None });
}
if low.contains("dns error")
|| low.contains("failed to lookup address")
|| low.contains("no address associated")
{
return Some(Failure { reason: "This address no longer resolves; the site is gone.", new_url: None });
}
None
}
/// True when a body is an OPML document rather than a feed.
///
/// The original matched on the URL ending in ".opml" (iPXClass.py:34), which misses an
@@ -117,17 +161,213 @@ pub fn opml_title(bytes: &[u8]) -> Option<String> {
.filter(|t| !t.is_empty())
}
/// The token and show of a Patreon feed link, or None for any other URL.
///
/// Patreon gives each patron one token per creator. With no show it stands for the creator,
/// whose feed carries every show at once.
fn patreon_parts(url: &str) -> Option<(String, Option<String>)> {
let u = url::Url::parse(url).ok()?;
if !matches!(u.host_str()?, "patreon.com" | "www.patreon.com") || !u.path().starts_with("/rss") {
return None;
}
let param = |name: &str| u.query_pairs().find(|(k, _)| k == name).map(|(_, v)| v.into_owned());
Some((param("auth")?, param("show")))
}
/// A Patreon link naming a creator but no show.
pub fn is_patreon_creator(url: &str) -> bool {
matches!(patreon_parts(url), Some((_, None)))
}
/// What was typed into Add feed, as a URL. A bare Patreon token is taken as its creator's
/// feed, since the token alone says whose it is.
pub fn expand_input(input: &str) -> String {
let s = input.trim();
let token = s.len() >= 20 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_');
if token { format!("https://www.patreon.com/rss?auth={s}") } else { s.to_owned() }
}
/// Whether two URLs are the same feed. One Patreon show has several spellings -- by the
/// creator's name, by number, or with no creator at all -- and the token and show are what
/// identify it.
pub fn same_feed(a: &str, b: &str) -> bool {
a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b))
}
/// A Patreon creator's name and shows, each show as (title, feed URL).
///
/// ponytail: Patreon's own web API, undocumented, asked without signing in. If it changes,
/// finding shows stops and the show feeds already found keep working. The documented API
/// needs an OAuth client per install and does not list shows.
pub async fn patreon_shows(
client: &reqwest::Client,
url: &str,
) -> Result<(Option<String>, Vec<(String, String)>)> {
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
let mut resp = client.get(url).send().await.context("connecting")?;
if !resp.status().is_success() {
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
}
let mut head = Vec::new();
while patreon_campaign(&head).is_none() && head.len() < 64 * 1024 {
let Some(chunk) = resp.chunk().await.context("reading the feed")? else { break };
head.extend_from_slice(&chunk);
}
let campaign = patreon_campaign(&head)
.ok_or_else(|| anyhow!("the Patreon feed does not say whose it is"))?;
let api = format!(
"https://www.patreon.com/api/campaigns/{campaign}\
?include=shows&fields%5Bcampaign%5D=name&fields%5Bcollection%5D=title"
);
let resp = client.get(api).send().await.context("asking Patreon for the shows")?;
if !resp.status().is_success() {
return Err(anyhow!("Patreon would not list the shows: HTTP {}", resp.status()));
}
let (name, shows) = parse_patreon_shows(&resp.bytes().await.context("reading the shows")?)?;
Ok((name, shows.into_iter().map(|(id, title)| (title, format!("{url}&show={id}"))).collect()))
}
/// The campaign number in the start of a Patreon feed.
fn patreon_campaign(head: &[u8]) -> Option<String> {
let text = String::from_utf8_lossy(head);
text.match_indices("patreon.com/rss/").find_map(|(i, m)| {
let id: String = text[i + m.len()..].chars().take_while(char::is_ascii_digit).collect();
(!id.is_empty()).then_some(id)
})
}
/// A campaign's name and its shows as (id, title), from Patreon's JSON:API answer.
fn parse_patreon_shows(json: &[u8]) -> Result<(Option<String>, Vec<(String, String)>)> {
let v: serde_json::Value = serde_json::from_slice(json).context("Patreon's answer is not JSON")?;
// Missing is not the same as none. Read as no shows, the creator feed would be scanned as
// a plain feed, claim every show's files, and leave the shows empty once the list returned.
let ids = v["data"]["relationships"]["shows"]["data"]
.as_array()
.ok_or_else(|| anyhow!("Patreon's answer does not list the shows"))?;
let title = |id: &str| -> Option<String> {
let show = v["included"].as_array()?.iter().find(|x| x["type"] == "collection" && x["id"] == id)?;
show["attributes"]["title"].as_str().map(|t| t.trim().to_owned())
};
let shows = ids
.iter()
.filter_map(|s| s["id"].as_str())
.map(|id| (id.to_owned(), title(id).unwrap_or_else(|| format!("Show {id}"))))
.collect();
Ok((v["data"]["attributes"]["name"].as_str().map(str::to_owned), shows))
}
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
match rss::Channel::read_from(bytes) {
Ok(ch) => Ok(from_rss(ch, bytes)),
Err(rss_err) => match atom_syndication::Feed::read_from(bytes) {
Ok(feed) => Ok(from_atom(feed)),
Err(atom_err) => Err(anyhow!("not RSS ({rss_err}) and not Atom ({atom_err})")),
Err(atom_err) => {
// Some publishers (kcpw, feedland) write a bare "&" in a URL instead of
// "&amp;". Strict XML parsers refuse it; browsers don't. Retry once with
// every offending "&" escaped rather than fail outright.
let escaped = escape_bare_ampersands(bytes);
if escaped != bytes {
if let Ok(ch) = rss::Channel::read_from(escaped.as_slice()) {
return Ok(from_rss(ch, &escaped));
}
if let Ok(feed) = atom_syndication::Feed::read_from(escaped.as_slice()) {
return Ok(from_atom(feed));
}
}
Err(match alternate_feed_link(bytes) {
Some(href) if looks_like_html(bytes) => {
anyhow!("got a web page, not a feed; it links {href} as its feed")
}
None if looks_like_html(bytes) => anyhow!("got a web page, not a feed"),
_ => anyhow!("not RSS ({rss_err}) and not Atom ({atom_err})"),
})
}
},
}
}
/// Whether a body is a web page rather than a feed: most of the errors traced back to a feed
/// that moved or a domain that lapsed, with the old URL now serving the site instead (or a
/// redirect to it). `is_opml` already sniffs the other "not actually a feed" case.
fn looks_like_html(bytes: &[u8]) -> bool {
let head = String::from_utf8_lossy(&bytes[..bytes.len().min(2048)]).to_lowercase();
head.contains("<!doctype html") || head.contains("<html")
}
/// The feed a web page names as its own via `<link rel="alternate" type="application/rss+xml"
/// href="...">` (or the Atom equivalent) -- how the new address was found for om.co, ms.now,
/// Letters of Note, the Daily Dot, Hell Gate, The Frame Lab and Daily Kos.
fn alternate_feed_link(bytes: &[u8]) -> Option<String> {
let text = String::from_utf8_lossy(bytes);
let lower = text.to_lowercase();
let mut pos = 0;
while let Some(rel) = lower[pos..].find("<link") {
let start = pos + rel;
let Some(end) = lower[start..].find('>').map(|e| start + e) else { break };
pos = end + 1;
let tag = &text[start..end];
let tag_lower = &lower[start..end];
let is_alternate = tag_lower.contains("rel=\"alternate\"") || tag_lower.contains("rel='alternate'");
let is_feed_type = tag_lower.contains("rss+xml") || tag_lower.contains("atom+xml");
if is_alternate && is_feed_type
&& let Some(href) = tag_attr(tag, "href")
{
return Some(href);
}
}
None
}
/// The value of one attribute in an HTML/XML start tag, however it is quoted.
fn tag_attr(tag: &str, name: &str) -> Option<String> {
let key = format!("{name}=");
let idx = tag.to_lowercase().find(&key)?;
let after = &tag[idx + key.len()..];
let quote = after.chars().next()?;
if quote != '"' && quote != '\'' {
return None;
}
let rest = &after[1..];
let close = rest.find(quote)?;
Some(rest[..close].trim().to_owned())
}
/// Escapes every `&` that does not already start a recognized XML entity
/// (`&amp;`, `&lt;`, `&gt;`, `&quot;`, `&apos;`, or a numeric reference like `&#39;`).
fn escape_bare_ampersands(bytes: &[u8]) -> Vec<u8> {
fn is_entity_start(rest: &[u8]) -> bool {
for named in [&b"amp;"[..], b"lt;", b"gt;", b"quot;", b"apos;"] {
if rest.starts_with(named) {
return true;
}
}
let digits = if rest.starts_with(b"#x") || rest.starts_with(b"#X") {
&rest[2..]
} else if rest.starts_with(b"#") {
&rest[1..]
} else {
return false;
};
let len = digits.iter().take_while(|b| b.is_ascii_alphanumeric()).count();
len > 0 && digits.get(len) == Some(&b';')
}
let mut out = Vec::with_capacity(bytes.len());
let mut i = 0;
while i < bytes.len() {
if bytes[i] == b'&' && !is_entity_start(&bytes[i + 1..]) {
out.extend_from_slice(b"&amp;");
} else {
out.push(bytes[i]);
}
i += 1;
}
out
}
/// Every `<enclosure>` of every `<item>`, in document order.
///
/// The `rss` crate models an item as having at most one enclosure -- which is what RSS 2.0
@@ -243,7 +483,7 @@ fn from_rss(ch: rss::Channel, bytes: &[u8]) -> ParsedFeed {
link: non_empty(item.link()),
published: item.pub_date().and_then(parse_date),
// Content wins over description, as __getEntries preferred entry.content.
description: non_empty(item.content()).or_else(|| non_empty(item.description())),
description: body(item.content(), item.description()),
categories: item
.categories()
.iter()
@@ -309,11 +549,7 @@ fn from_atom(feed: atom_syndication::Feed) -> ParsedFeed {
title: non_empty(Some(e.title().as_str())),
link: alt.map(str::to_owned),
published: e.published().or(Some(e.updated())).map(|d| d.timestamp()),
description: e
.content()
.and_then(|c| c.value())
.or_else(|| e.summary().map(|s| s.as_str()))
.map(str::to_owned),
description: body(e.content().and_then(|c| c.value()), e.summary().map(|s| s.as_str())),
categories: e.categories().iter().map(|c| c.term().to_owned()).collect(),
explicit: false,
image: None,
@@ -358,6 +594,25 @@ fn non_empty(s: Option<&str>) -> Option<String> {
s.map(str::trim).filter(|s| !s.is_empty()).map(str::to_owned)
}
/// An item's show notes: its full body when that is whole, else its description.
///
/// libsyn served Daily Meditation Podcast's `content:encoded` cut at the `>` inside a class name
/// pasted from a web app (`[&:has([data-writing-block])>*]:pointer-events-auto`), so the body
/// began halfway through a tag and the page showed the rest of the tag as text. The same item's
/// `description` was whole. With no description to fall back on, a damaged body beats none.
fn body(content: Option<&str>, description: Option<&str>) -> Option<String> {
non_empty(content)
.filter(|c| !starts_mid_tag(c))
.or_else(|| non_empty(description))
.or_else(|| non_empty(content))
}
/// Text that closes an attribute list (`">`) before any tag has opened is the tail of a tag whose
/// start was cut off.
fn starts_mid_tag(html: &str) -> bool {
html[..html.find('<').unwrap_or(html.len())].contains("\">")
}
/// The picture to show beside an item, in order of how deliberate it is:
/// `itunes:image`, then Media RSS `media:thumbnail`, then a `media:content` that is an
/// image, and finally an image enclosure -- which is how a blog's article picture arrives
@@ -451,6 +706,18 @@ mod tests {
);
}
#[test]
fn a_body_cut_off_mid_tag_gives_way_to_the_description() {
// How libsyn served Daily Meditation Podcast #3477: content:encoded began inside a tag.
let cut = r#"*]:pointer-events-auto R6Vx5W_threadScrollVars" dir="auto" data-turn="assistant"> <p>What if</p>"#;
let whole = r#"<div class="[&:has([data-writing-block])>*]:pointer-events-auto"><p>What if</p></div>"#;
assert_eq!(body(Some(cut), Some(whole)).as_deref(), Some(whole));
assert_eq!(body(Some("<p>Notes</p>"), Some("Summary")).as_deref(), Some("<p>Notes</p>"), "a whole body wins");
assert_eq!(body(Some("Plain notes, no tags."), Some("Summary")).as_deref(), Some("Plain notes, no tags."));
assert_eq!(body(Some(cut), None).as_deref(), Some(cut), "a damaged body beats none");
assert_eq!(body(None, Some("Summary")).as_deref(), Some("Summary"));
}
#[test]
fn feed_level_explicit_overrides_entries() {
let xml = br#"<?xml version="1.0"?>
@@ -467,6 +734,46 @@ mod tests {
);
}
#[test]
fn explain_failure_translates_the_errors_the_ui_should_flag() {
assert_eq!(
explain_failure("HTTP 404 Not Found").unwrap().reason,
"The publisher took this feed down, or moved it."
);
assert_eq!(explain_failure("HTTP 401 Unauthorized").unwrap().reason, "The site refuses ipx's requests.");
assert_eq!(explain_failure("HTTP 403 Forbidden").unwrap().reason, "The site refuses ipx's requests.");
assert_eq!(explain_failure("HTTP 402 Payment Required").unwrap().reason, "The feed now needs a paid plan.");
let dns = explain_failure("connecting: dns error: failed to lookup address information").unwrap();
assert_eq!(dns.reason, "This address no longer resolves; the site is gone.");
let moved = explain_failure("got a web page, not a feed; it links https://x/feed as its feed").unwrap();
assert_eq!(moved.new_url.as_deref(), Some("https://x/feed"));
assert!(explain_failure("got a web page, not a feed").unwrap().new_url.is_none());
for transient in ["HTTP 500 Internal Server Error", "HTTP 429 Too Many Requests", "operation timed out"] {
assert!(explain_failure(transient).is_none(), "{transient} must not be flagged");
}
}
#[test]
fn a_web_page_says_so_and_names_the_feed_it_links() {
let html = br#"<!doctype html><html><head>
<link rel="alternate" type="application/rss+xml" href="https://x.example/feed">
</head><body>not a feed</body></html>"#;
let err = parse(html).unwrap_err().to_string();
assert_eq!(err, "got a web page, not a feed; it links https://x.example/feed as its feed");
}
#[test]
fn a_web_page_with_no_feed_link_still_says_so() {
let html = b"<!doctype html><html><body>moved</body></html>";
assert_eq!(parse(html).unwrap_err().to_string(), "got a web page, not a feed");
}
#[test]
fn garbage_that_is_not_html_gets_the_original_parser_errors() {
let err = parse(b"not xml at all").unwrap_err().to_string();
assert!(err.starts_with("not RSS ("), "{err}");
}
#[test]
fn parses_atom_enclosure_links() {
let bytes = include_bytes!("../tests/data/atom.xml");
@@ -489,6 +796,27 @@ mod tests {
);
}
#[test]
fn a_bare_ampersand_in_a_link_is_repaired_and_parsed() {
// kcpw.org: <link>https://kcpw.org/?post_type=post&p=125715</link> -- a bare "&"
// that strict XML rejects but browsers accept.
let xml = br#"<?xml version="1.0"?>
<rss version="2.0"><channel><title>X</title><link>https://x</link><description>d</description>
<item><title>a</title><guid>g1</guid>
<link>https://kcpw.org/?post_type=post&p=125715</link>
<enclosure url="https://x/a.mp3?a=1&b=2" length="1" type="audio/mpeg"/></item>
</channel></rss>"#;
let feed = parse(xml).unwrap();
assert_eq!(feed.entries[0].link.as_deref(), Some("https://kcpw.org/?post_type=post&p=125715"));
assert_eq!(feed.entries[0].enclosures[0].url, "https://x/a.mp3?a=1&b=2");
}
#[test]
fn escape_bare_ampersands_leaves_real_entities_alone() {
let out = escape_bare_ampersands(b"a&amp;b &lt;x&gt; &#39; &#x2F; c&d");
assert_eq!(out, b"a&amp;b &lt;x&gt; &#39; &#x2F; c&amp;d");
}
#[test]
fn the_rss_title_always_wins_and_episode_numbers_stay_metadata() {
// Some feeds set a different itunes:title. The displayed title is always the RSS
@@ -550,6 +878,45 @@ mod tests {
assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
}
#[test]
fn a_patreon_creator_is_a_list_of_its_shows() {
let tok = "AbCdEfGhIjKlMnOpQrStUvWxYz012_-9";
assert_eq!(expand_input(&format!(" {tok} ")), format!("https://www.patreon.com/rss?auth={tok}"));
assert_eq!(expand_input("https://example.com/rss"), "https://example.com/rss");
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss/glasscannon?auth={tok}")));
assert!(is_patreon_creator(&format!("https://www.patreon.com/rss?auth={tok}")));
assert!(!is_patreon_creator(&format!("https://www.patreon.com/rss/x?auth={tok}&show=1")), "one show is a feed");
assert!(!is_patreon_creator(&format!("https://example.com/rss?auth={tok}")));
// The show you already have by name is the one a bare token would add by number.
assert!(same_feed(
&format!("https://www.patreon.com/rss/glasscannon?auth={tok}&show=2073588"),
&format!("https://www.patreon.com/rss?auth={tok}&show=2073588"),
));
assert!(!same_feed(
&format!("https://www.patreon.com/rss?auth={tok}&show=1"),
&format!("https://www.patreon.com/rss?auth={tok}&show=2"),
));
// The self link carries the campaign by number, whichever spelling was asked for.
let head = br#"<rss><channel><link>https://www.patreon.com/glasscannon</link>
<atom:link href="https://www.patreon.com/rss/369921?auth=t" rel="self"/>"#;
assert_eq!(patreon_campaign(head).as_deref(), Some("369921"));
assert_eq!(patreon_campaign(b"<rss><channel><title>T"), None);
let json = br#"{"data":{"id":"369921","type":"campaign","attributes":{"name":"The Glass Cannon Network"},
"relationships":{"shows":{"data":[{"id":"2073588","type":"collection"},{"id":"2073636","type":"collection"}]}}},
"included":[{"id":"2073588","type":"collection","attributes":{"title":"Get in the Trunk "}},
{"id":"2073636","type":"collection","attributes":{"title":"Shadowdark"}}]}"#;
let (name, shows) = parse_patreon_shows(json).unwrap();
assert_eq!(name.as_deref(), Some("The Glass Cannon Network"));
assert_eq!(shows, [("2073588".into(), "Get in the Trunk".into()), ("2073636".into(), "Shadowdark".into())]);
// An answer that stops naming the shows is an error, never "this creator has none".
assert!(parse_patreon_shows(br#"{"data":{"attributes":{"name":"X"}}}"#).is_err());
}
#[test]
fn an_item_may_carry_several_enclosures() {
// The rss crate keeps only one per item -- the last -- so these come from the XML.

View File

@@ -172,11 +172,18 @@ pub async fn daemon_is_live(path: &Path) -> bool {
UnixStream::connect(path).await.is_ok()
}
/// Answers `status` for the socket, without the worker. The worker runs one job at a time, and a
/// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon
/// dead. The answer goes to the client that asked and no one else: broadcast, it ended any
/// `ipx fetch` that was watching a scan, since `status` is a terminal event.
pub type StatusFn = std::sync::Arc<dyn Fn() -> Event + Send + Sync>;
/// Accepts connections, feeding commands to `cmds` and events from `events` back out.
pub async fn serve(
path: PathBuf,
events: broadcast::Sender<Event>,
cmds: mpsc::Sender<Command>,
status: StatusFn,
) -> Result<()> {
// A socket file left by a crashed daemon would block the bind; a live one was already
// rejected by the caller's daemon_is_live() check.
@@ -195,8 +202,9 @@ pub async fn serve(
let (stream, _) = listener.accept().await?;
let rx = events.subscribe();
let cmds = cmds.clone();
let status = status.clone();
tokio::spawn(async move {
if let Err(e) = handle(stream, rx, cmds).await {
if let Err(e) = handle(stream, rx, cmds, status).await {
tracing::debug!(error = %e, "client gone");
}
});
@@ -207,12 +215,21 @@ async fn handle(
stream: UnixStream,
mut rx: broadcast::Receiver<Event>,
cmds: mpsc::Sender<Command>,
status: StatusFn,
) -> Result<()> {
let (read, mut write) = stream.into_split();
// Events out.
// Events out: everything broadcast, and the answers meant for this client alone.
let (reply, mut replies) = mpsc::channel::<Event>(4);
let writer = tokio::spawn(async move {
while let Ok(ev) = rx.recv().await {
loop {
let ev = tokio::select! {
Some(ev) = replies.recv() => ev,
got = rx.recv() => match got {
Ok(ev) => ev,
Err(_) => break,
},
};
let mut line = serde_json::to_string(&ev).unwrap_or_default();
line.push('\n');
if write.write_all(line.as_bytes()).await.is_err() {
@@ -229,6 +246,15 @@ async fn handle(
continue;
}
match serde_json::from_str::<Command>(line) {
// Answered here, not queued behind whatever the worker is on: see StatusFn.
Ok(Command::Status) => {
tracing::info!(target: "ipx::io", "-> {line}");
let ev = status();
if let Ok(json) = serde_json::to_string(&ev) {
tracing::info!(target: "ipx::io", "<- {json}");
}
let _ = reply.send(ev).await;
}
Ok(cmd) => {
if cmds.send(cmd).await.is_err() {
break; // Worker is gone; so are we.
@@ -329,4 +355,29 @@ mod tests {
}
.is_terminal());
}
#[tokio::test]
async fn status_is_answered_while_the_worker_is_busy() {
// The queue is full and nobody drains it, as when the worker is deep in a long download:
// anything sent to it would wait for ever.
let (cmds, _worker) = mpsc::channel::<Command>(1);
cmds.send(Command::Reap { dry_run: true }).await.unwrap();
let (events, _) = broadcast::channel::<Event>(8);
// Another client, watching a scan: it must not be handed someone else's answer, which
// would end its session.
let mut watcher = events.subscribe();
let status: StatusFn = std::sync::Arc::new(|| Event::Status { feeds: 1, pending: 2, downloaded: 3 });
let (client, server) = UnixStream::pair().unwrap();
tokio::spawn(handle(server, events.subscribe(), cmds, status));
let (read, mut write) = client.into_split();
write.write_all(b"{\"cmd\":\"status\"}\n").await.unwrap();
let line = tokio::time::timeout(std::time::Duration::from_secs(2), BufReader::new(read).lines().next_line())
.await
.expect("status waited behind the worker")
.unwrap()
.unwrap();
assert!(line.contains(r#""ev":"status""#), "{line}");
assert!(watcher.try_recv().is_err(), "the answer went to every client, not just the one asking");
}
}

View File

@@ -111,18 +111,11 @@ impl Visit for Collect {
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
self.add(field, format!("{value:?}"));
}
// Numbers and bools reach record_debug through the trait's defaults, which prints them the
// same way. A string would print quoted there, hence its own method.
fn record_str(&mut self, field: &Field, value: &str) {
self.add(field, value.to_owned());
}
fn record_i64(&mut self, field: &Field, value: i64) {
self.add(field, value.to_string());
}
fn record_u64(&mut self, field: &Field, value: u64) {
self.add(field, value.to_string());
}
fn record_bool(&mut self, field: &Field, value: bool) {
self.add(field, value.to_string());
}
}
#[cfg(test)]

View File

@@ -99,6 +99,9 @@ enum UserCmd {
Passwd { name: String },
/// Delete an account and everything it knows: its subscriptions and read state
Rm { name: String },
/// Rename an account, keeping its feeds, read state and admin rights. This is how an
/// account made before the proxy takes the name the proxy signs it in as
Rename { name: String, new_name: String },
}
/// What a brand new database starts with, so there is always a way in. Announced loudly
@@ -273,11 +276,16 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
println!("no accounts yet: ipx user add <name>");
}
for u in users {
let added = u
.created
.and_then(|t| chrono::DateTime::from_timestamp(t, 0))
.map_or("?".into(), |d| d.format("%Y-%m-%d").to_string());
let seen = u.last_login.map_or("never signed in".into(), |t| format!("signed in {}", ago(Some(t))));
println!(
"{:<20} {:<8} {}",
"{:<20} {:<6} {:<11} added {added} {seen}",
u.name,
if u.is_admin { "admin" } else { "" },
if u.pass_hash.is_some() { "password" } else { "proxy only" }
if u.pass_hash.is_some() { "password" } else { "proxy only" },
);
}
Ok(())
@@ -292,6 +300,22 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
println!("password changed for {name}");
Ok(())
}
UserCmd::Rename { name, new_name } => {
let name = name.trim().to_ascii_lowercase();
// The same rules as a name the proxy vouches for, or the proxy would never find it.
let new_name = crate::auth::name_from_header(&new_name)
.ok_or_else(|| anyhow::anyhow!("not a usable name: no commas, semicolons or line breaks"))?;
let user = ctx
.db
.user_by_name(&name)?
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
if ctx.db.user_by_name(&new_name)?.is_some() {
anyhow::bail!("{new_name} already exists");
}
ctx.db.rename_user(user.id, &new_name)?;
println!("renamed {name} to {new_name}");
Ok(())
}
UserCmd::Rm { name } => {
let name = name.trim().to_ascii_lowercase();
let user = ctx
@@ -315,14 +339,24 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
Cmd::Reap { dry_run } => reap(ctx, dry_run, true),
Cmd::Download { enclosure } => download_one(ctx, enclosure).await,
Cmd::Status => {
let (pending, downloaded) = ctx.db.counts()?;
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0);
ctx.out.emit(Event::Status { feeds, pending, downloaded });
ctx.out.emit(status(ctx));
Ok(())
}
}
}
/// The counts `ipx status` prints. A running daemon's socket answers with this directly rather
/// than through the job queue.
fn status(ctx: &Ctx) -> Event {
match ctx.db.counts() {
Ok((pending, downloaded)) => {
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0);
Event::Status { feeds, pending, downloaded }
}
Err(e) => Event::Error { msg: format!("{e:#}") },
}
}
async fn daemon(
ctx: Arc<Ctx>,
config_path: PathBuf,
@@ -334,8 +368,7 @@ async fn daemon(
anyhow::bail!("a daemon is already listening on {}", socket.display());
}
// A database with nobody in it cannot be signed into, and an install that predates
// accounts still has to serve its owner. Both get the same starting point.
// A database with nobody in it cannot be signed into.
if ctx.db.users()?.is_empty() {
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
tracing::warn!(
@@ -344,22 +377,15 @@ async fn daemon(
);
}
// A library that predates accounts belongs to whoever was using it: the admin.
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
match ctx.db.adopt_existing_library(admin.id, &catalogue) {
match ctx.db.adopt_catalogue(admin.id, &catalogue) {
Ok(0) => {}
Ok(n) => tracing::info!(user = %admin.name, entries = n, "adopted the existing library"),
Err(e) => tracing::error!(error = %e, "could not adopt the existing library"),
Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
}
}
match migrate_opml_children(&ctx) {
Ok(n) if n > 0 => tracing::info!(count = n, "moved OPML feeds out of config.toml into the database"),
Ok(_) => {}
Err(e) => tracing::warn!(error = ?e, "could not tidy OPML feeds out of the config"),
}
match ctx.db.requeue_interrupted() {
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
Ok(_) => {}
@@ -369,7 +395,12 @@ async fn daemon(
let (tx_cmd, mut rx_cmd) = mpsc::channel::<Cmd>(64);
let web = start_web(&ctx, &config_path, web_addr, &tx_cmd, &events).await?;
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd));
// status is answered by the socket itself; everything else waits its turn in the queue.
let answer: ipc::StatusFn = {
let ctx = ctx.clone();
Arc::new(move || status(&ctx))
};
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer));
// One command at a time: the queue is what keeps two scans from overlapping.
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
@@ -468,7 +499,7 @@ async fn start_web(
let mut fresh = (*cfg).clone();
fresh.web.enabled = true;
fresh.web.bind = bind.clone();
fresh.web.token = web::generate_token();
fresh.web.token = crate::auth::new_session_token();
fresh.save(config_path)?;
ctx.reload_cfg(config_path)?;
println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token);
@@ -517,8 +548,9 @@ async fn add(
keywords: Vec<String>,
) -> Result<()> {
let mut cfg = (*ctx.cfg()).clone();
let url = &feed::expand_input(url);
// Includes feeds derived from an OPML, or the same show could be added twice.
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| s.cfg.url == url) {
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
anyhow::bail!("already subscribed as {:?}", existing.id);
}
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
@@ -569,10 +601,17 @@ pub async fn add_one(
// Slugs must be unique across derived feeds too, or a new feed can collide with one
// an OPML already introduced.
let taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
.into_iter()
.map(|s| (s.id, s.cfg))
.collect();
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
// it gets its history back, and a different feed does not inherit someone else's.
for (id, other) in ctx.db.feed_urls()? {
if !feed::same_feed(&other, url) {
taken.entry(id).or_insert_with(|| probe.clone());
}
}
let id = config::unique_slug(&title, &taken);
cfg.feeds.insert(id.clone(), probe);
Ok(id)
@@ -598,6 +637,7 @@ fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> {
cfg.save(config_path)?;
// State and files stay: re-adding the feed should not re-download its back catalogue.
println!("removed {feed}; downloads and history kept");
retire_group(ctx, feed)?;
Ok(())
}
@@ -803,11 +843,15 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
feed: id.clone(),
reason: "not modified".into(),
}),
Ok(Outcome::Empty) => ctx.out.emit(Event::FeedSkip {
feed: id.clone(),
reason: "nothing yet".into(),
}),
Ok(Outcome::Opml { added, removed, kept, total }) => {
ctx.out.emit(Event::FeedSkip {
feed: id.clone(),
reason: format!(
"OPML: {total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
"{total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing",
added.len()
),
});
@@ -880,6 +924,12 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
continue; // promoted to config at some point; that entry wins
}
let parent = cfg.feeds.get(&m.group_id);
if parent.is_none() {
// The OPML or Patreon feed this was derived from is no longer in config --
// removing it should have retired these rows too (see `retire_group`), but
// skip them here regardless so a row that slips through is never scanned.
continue;
}
let base = parent
.and_then(|p| p.folder.clone())
.or_else(|| ctx.db.feed_summary(&m.group_id).ok().and_then(|s| s.title))
@@ -908,34 +958,20 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
Ok(out)
}
/// Moves OPML children that older versions wrote into config.toml over to the database.
/// They were never yours to edit, and 80-odd of them made the file unreadable.
fn migrate_opml_children(ctx: &Ctx) -> Result<usize> {
let cfg = (*ctx.cfg()).clone();
let children: Vec<(String, config::Feed)> = cfg
.feeds
.iter()
.filter(|(_, f)| f.group.is_some())
.map(|(id, f)| (id.clone(), f.clone()))
.collect();
if children.is_empty() {
return Ok(0);
/// Retires every feed derived from `parent_id`, now that nothing subscribes to the OPML or
/// Patreon feed that listed them: the same rule `sync_group` applies to one the list drops --
/// removed if nothing was downloaded, orphaned and kept otherwise. Called once the parent
/// itself is removed, since `subscriptions()` would otherwise keep scanning them under a
/// fallback policy meant for a feed with no parent at all.
pub fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
for m in ctx.db.managed_feeds()?.into_iter().filter(|m| m.group_id == parent_id) {
if ctx.db.downloaded_count(&m.id).unwrap_or(1) > 0 {
ctx.db.set_orphaned(&m.id, true)?;
} else {
ctx.db.drop_managed(&m.id)?;
}
}
let mut fresh = cfg.clone();
for (id, f) in &children {
let group = f.group.clone().unwrap_or_default();
let title = ctx
.db
.feed_summary(id)
.ok()
.and_then(|s| s.title)
.unwrap_or_else(|| id.clone());
ctx.db.upsert_managed(id, &f.url, &title, &group)?;
fresh.feeds.remove(id);
}
fresh.save(&ctx.config_path)?;
ctx.reload_cfg(&ctx.config_path)?;
Ok(children.len())
Ok(())
}
/// Seconds to wait before re-checking a feed.
@@ -962,8 +998,11 @@ struct Scan {
/// What a scan of one feed turned out to be.
enum Outcome {
NotModified,
/// A response with nothing in it -- the British Antarctic Survey answers a 202 with an
/// empty body when it has nothing new to publish. Not a parse failure; try again later.
Empty,
Feed(Scan),
/// The URL served an OPML document, so it is a subscription list rather than a feed.
/// The URL is a list of feeds rather than a feed: an OPML, or a Patreon creator's shows.
Opml { added: Vec<String>, removed: usize, kept: usize, total: usize },
}
@@ -973,6 +1012,31 @@ async fn scan_one(
feed_cfg: &config::Feed,
state: &db::HttpState,
) -> Result<Outcome> {
// A Patreon creator with more than one show is a list of feeds, like an OPML.
if feed::is_patreon_creator(&feed_cfg.url) {
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
Ok((name, shows)) if shows.len() > 1 => {
ctx.db.touch_feed(id, &feed_cfg.url)?;
if let Some(name) = name {
ctx.db.set_title(id, &name)?;
}
// Read as one feed before it was split, it listed every show's items in one
// heap. The items go; its files and read state move to each show as the show
// lists them (`Db::adopt`), so no show comes up empty for want of a URL.
ctx.db.clear_entries(id)?;
return sync_group(ctx, id, feed_cfg, &shows).await;
}
Ok(_) => {} // One show: the creator's feed is that show.
// Already split: keep the shows it has rather than read the creator as one heap.
Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e),
Err(e) => tracing::warn!(
feed = id,
error = %format!("{e:#}"),
"could not list the Patreon shows; reading it as one feed"
),
}
}
let mut fetched = feed::fetch(
&ctx.client,
feed_cfg,
@@ -999,6 +1063,11 @@ async fn scan_one(
feed::Fetched::Body { bytes, etag, last_modified } => (bytes, etag, last_modified),
};
if bytes.iter().all(u8::is_ascii_whitespace) {
ctx.db.touch_feed(id, &feed_cfg.url)?;
return Ok(Outcome::Empty);
}
// A subscribed OPML is a list of feeds, not a feed. The original matched on a ".opml"
// URL; sniffing the body also catches one served from a URL without that extension.
if feed::is_opml(&bytes) {
@@ -1018,19 +1087,38 @@ async fn scan_one(
)?;
let policy = policy_for(ctx, id, feed_cfg)?;
if let Some(parent) = &feed_cfg.group {
let listed: Vec<(&str, &str)> = parsed
.entries
.iter()
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
.collect();
ctx.db.adopt(parent, id, &listed)?;
}
// Verdicts are recorded in `state`, so the download queue below is just "everything still
// pending". A filter's verdict is looked at again on every scan, though: made once, at
// discovery, it outlived the setting behind it, and allowing explicit items afterwards
// changed nothing however often the feed was scanned.
let skipped = ctx.db.skipped_by_filter(id)?;
let mut scan = Scan::default();
for entry in &parsed.entries {
if ctx.db.record_entry(id, entry)? {
scan.new_entries += 1;
}
for enc in &entry.enclosures {
if !ctx.db.record_enclosure(id, &entry.guid, enc)? {
continue; // Seen before: downloaded, skipped or deliberately reaped.
}
// Filters run once, at discovery, and are recorded in `state`. The download
// queue below is then just "everything still pending".
if let Some(reason) = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc) {
ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?;
let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? {
None
} else if let Some(reason) = skipped.get(&enc.url) {
Some(reason.as_str())
} else {
continue; // Settled: queued, downloaded, reaped, or another feed's file.
};
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
if now != was {
match now {
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?,
None => ctx.db.mark_enclosure(&enc.url, "pending", None)?,
}
}
}
}
@@ -1114,12 +1202,6 @@ async fn scan_one(
Ok(Outcome::Feed(scan))
}
/// Brings the feed list in step with a subscribed OPML.
///
/// New entries are added under the OPML's group and folder. An entry that has gone from
/// the OPML is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
/// the UI to explain them.
async fn sync_opml(
ctx: &Arc<Ctx>,
parent_id: &str,
@@ -1130,25 +1212,44 @@ async fn sync_opml(
if let Some(title) = feed::opml_title(bytes) {
ctx.db.set_title(parent_id, &title)?;
}
sync_group(ctx, parent_id, parent, &listed).await
}
/// Brings the feed list in step with a list of feeds: a subscribed OPML, or a Patreon
/// creator's shows.
///
/// New entries are added under the list's group and folder. An entry that has gone from
/// the list is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it
/// is kept and flagged, because dropping it would orphan files on disk with nothing in
/// the UI to explain them.
async fn sync_group(
ctx: &Arc<Ctx>,
parent_id: &str,
parent: &config::Feed,
listed: &[(String, String)],
) -> Result<Outcome> {
let cfg = ctx.cfg();
let existing = ctx.db.managed_feeds()?;
let mut added = vec![];
for (title, url) in &listed {
for (title, url) in listed {
// Already known, whether derived or promoted into the config.
if let Some(m) = existing.iter().find(|m| &m.url == url) {
ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
continue;
}
if cfg.feeds.values().any(|f| &f.url == url) {
// A Patreon show you added by hand may be spelled differently from the one listed.
if cfg.feeds.values().any(|f| feed::same_feed(&f.url, url)) {
continue;
}
// A removed feed keeps its rows, so its id is only free again for the same feed.
let known = ctx.db.feed_urls()?;
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
.feeds
.keys()
.chain(existing.iter().map(|m| &m.id))
.chain(added.iter())
.chain(known.iter().filter(|(_, u)| !feed::same_feed(u, url)).map(|(id, _)| id))
.map(|id| (id.clone(), parent.clone()))
.collect();
let id = config::unique_slug(title, &taken);
@@ -1255,7 +1356,7 @@ pub struct Policy {
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
let global = ctx.cfg().general.max_new_per_check;
Ok(merge_policy(&ctx.db.subscribers(id)?, feed_cfg, global))
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global))
}
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
@@ -1569,4 +1670,46 @@ mod tests {
let p = merge_policy(&[sub(None, Some(false), None), sub(None, Some(true), None)], &feed(), 3);
assert!(p.auto_download);
}
fn test_ctx(cfg: config::Config) -> Ctx {
Ctx {
cfg: std::sync::RwLock::new(Arc::new(cfg)),
db: db::Db::memory().unwrap(),
client: reqwest::Client::new(),
out: Emitter::terminal(),
torrents: tokio::sync::OnceCell::new(),
torrent_slots: Arc::new(tokio::sync::Semaphore::new(2)),
config_path: PathBuf::new(),
detach_torrents: false,
}
}
#[test]
fn a_derived_feed_is_not_scanned_once_its_opml_leaves_config() {
// davewiner: the OPML subscription left config.toml, but its 922 derived rows
// stayed in the database and kept being scanned under the no-parent fallback.
let ctx = test_ctx(config::Config::default());
ctx.db.upsert_managed("child", "http://x/child.xml", "Child", "gone-opml").unwrap();
assert!(
subscriptions(&ctx).unwrap().iter().all(|s| s.id != "child"),
"a derived feed whose parent is gone from config must not be scanned"
);
}
#[test]
fn retiring_a_group_drops_what_was_never_downloaded_and_orphans_the_rest() {
let ctx = test_ctx(config::Config::default());
ctx.db.upsert_managed("empty", "http://x/empty.xml", "Empty", "parent").unwrap();
ctx.db.upsert_managed("has-file", "http://x/has-file.xml", "Has File", "parent").unwrap();
let enc = feed::Enclosure { url: "http://x/ep.mp3".into(), mime: None, length: None };
ctx.db.record_enclosure("has-file", "g1", &enc).unwrap();
ctx.db.mark_downloaded(&enc.url, std::path::Path::new("/downloads/ep.mp3"), 1).unwrap();
retire_group(&ctx, "parent").unwrap();
let managed = ctx.db.managed_feeds().unwrap();
assert!(!managed.iter().any(|m| m.id == "empty"), "nothing downloaded, so it is forgotten");
assert!(managed.iter().any(|m| m.id == "has-file"), "has a file on disk, so it is kept");
assert!(ctx.db.feed_summary("has-file").unwrap().orphaned, "and flagged as orphaned");
}
}

View File

@@ -154,8 +154,8 @@ mod tests {
// One file serves both subscribers, so it takes both of them to release it.
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0);
INSERT INTO subscriptions (user_id, feed_id, created) VALUES (1,'f',0),(2,'f',0);
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f');
INSERT INTO entries (feed_id, guid, first_seen) VALUES
('f', 'keep', 0),
('f', 'half', 0),
@@ -189,7 +189,7 @@ mod tests {
fn prune_keeps_entries_that_still_have_a_file() {
let db = Db::memory().unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0);
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',1);
INSERT INTO entries (feed_id, guid, first_seen) VALUES
('f', 'has-file', 100),

View File

@@ -12,9 +12,7 @@ use axum::{
},
routing::{delete, get, patch, post},
};
use futures_util::StreamExt;
use serde::Deserialize;
use tokio_stream::wrappers::BroadcastStream;
use tower::ServiceExt;
use tower_http::services::ServeFile;
use serde::Serialize;
@@ -35,28 +33,6 @@ pub struct WebState {
pub events: broadcast::Sender<Event>,
}
/// A 32-hex-character shared secret, generated when config.toml has none.
///
/// ponytail: /dev/urandom rather than a CSPRNG crate -- 16 bytes, once, on a Unix-only
/// binary. Falls back to the clock only if urandom is somehow unreadable, which would be a
/// weak token, so that case is logged loudly.
pub fn generate_token() -> String {
use std::io::Read;
let mut bytes = [0u8; 16];
match std::fs::File::open("/dev/urandom").and_then(|mut f| f.read_exact(&mut bytes)) {
Ok(()) => {}
Err(e) => {
tracing::error!(error = %e, "could not read /dev/urandom; token is NOT secure");
let n = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as u64)
.unwrap_or(0);
bytes[..8].copy_from_slice(&n.to_le_bytes());
}
}
bytes.iter().map(|b| format!("{b:02x}")).collect()
}
pub fn router(state: WebState) -> Router {
Router::new()
.route("/", get(index))
@@ -88,6 +64,7 @@ pub fn router(state: WebState) -> Router {
// Signing in cannot require being signed in, so these sit outside the auth layer.
.route("/login", get(login_page))
.route("/api/login", post(login))
.route("/icon.png", get(icon))
.layer(middleware::from_fn(access_log))
.with_state(state)
}
@@ -114,23 +91,8 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
let cfg = state.ctx.cfg();
let token = cfg.web.token.clone();
let peer = req
.extensions()
.get::<axum::extract::ConnectInfo<std::net::SocketAddr>>()
.map(|c| c.0.ip().to_string())
.unwrap_or_default();
// 1. A header, but only from a hop we were told to believe. Anyone able to reach the
// port could otherwise send it and be whoever they liked.
let vouched = (!cfg.web.trusted_header.is_empty()
&& cfg.web.trusted_proxies.iter().any(|p| p == &peer))
.then(|| {
req.headers()
.get(&cfg.web.trusted_header)
.and_then(|v| v.to_str().ok())
.and_then(crate::auth::name_from_header)
})
.flatten();
// 1. A header, but only from a hop we were told to believe.
let vouched = vouched_name(&cfg, &req);
let mut set_cookie: Option<String> = None;
let mut user = None;
@@ -156,7 +118,13 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
None
}
};
// Every request comes vouched for; signed_in keeps one an hour. Failing to note the time
// must not turn anyone away, so its error goes unanswered.
if let Some(u) = &user {
let _ = state.ctx.db.signed_in(u.id);
}
}
let by_proxy = user.is_some();
// 2. A session cookie from signing in here.
if user.is_none() {
@@ -180,6 +148,10 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
if supplied.is_some_and(|t| constant_time_eq(&t, &token)) {
user = admin_user(&state);
if from_query.is_some() {
// The token link is a sign-in; the cookie it leaves behind is not one each time.
if let Some(u) = &user {
let _ = state.ctx.db.signed_in(u.id);
}
set_cookie = Some(format!(
"{COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000"
));
@@ -203,6 +175,7 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
};
req.extensions_mut().insert(user);
req.extensions_mut().insert(Proxied(by_proxy));
let mut resp = next.run(req).await;
if let Some(c) = set_cookie {
if let Ok(v) = header::HeaderValue::from_str(&c) {
@@ -214,6 +187,29 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
const SESSION_COOKIE: &str = "ipx_session";
/// Whether the proxy signed this request in, rather than a session or the token: signing out
/// has to go through the proxy then, or its next request signs the person straight back in.
#[derive(Clone, Copy)]
struct Proxied(bool);
/// The name the proxy vouches for, when this request came from one of `trusted_proxies` and
/// carries `trusted_header`. Anyone able to reach the port could otherwise send the header and
/// be whoever they liked.
fn vouched_name(cfg: &crate::config::Config, req: &Request) -> Option<String> {
let peer = req
.extensions()
.get::<axum::extract::ConnectInfo<std::net::SocketAddr>>()
.map(|c| c.0.ip().to_string())
.unwrap_or_default();
if cfg.web.trusted_header.is_empty() || !cfg.web.trusted_proxies.iter().any(|p| p == &peer) {
return None;
}
req.headers()
.get(&cfg.web.trusted_header)
.and_then(|v| v.to_str().ok())
.and_then(crate::auth::name_from_header)
}
/// Handlers take `User` to say they need one; the auth layer put it there, and nothing
/// reaches a handler without passing through it.
impl<S: Send + Sync> axum::extract::FromRequestParts<S> for crate::db::User {
@@ -276,6 +272,7 @@ async fn login(
let user = user.expect("verified above");
let token = crate::auth::new_session_token();
state.ctx.db.create_session(user.id, &token)?;
state.ctx.db.signed_in(user.id)?;
tracing::info!(user = %user.name, "signed in");
let days = state.ctx.cfg().web.session_days.max(1);
@@ -306,8 +303,15 @@ async fn logout(State(state): State<WebState>, req: Request) -> Response {
resp
}
async fn me(user: crate::db::User) -> Json<serde_json::Value> {
Json(serde_json::json!({ "name": user.name, "admin": user.is_admin }))
/// Who is signed in, and, for someone the proxy signed in, where Sign out should send them.
async fn me(
State(state): State<WebState>,
user: crate::db::User,
axum::Extension(Proxied(by_proxy)): axum::Extension<Proxied>,
) -> Json<serde_json::Value> {
let url = state.ctx.cfg().web.sign_out_url.clone();
let sign_out = (by_proxy && !url.is_empty()).then_some(url);
Json(serde_json::json!({ "name": user.name, "admin": user.is_admin, "sign_out": sign_out }))
}
// ---- accounts: admin only ----
@@ -340,6 +344,7 @@ async fn list_users(
.map(|u| {
serde_json::json!({
"id": u.id, "name": u.name, "admin": u.is_admin, "password": u.pass_hash.is_some(),
"created": u.created, "last_login": u.last_login,
})
})
.collect();
@@ -429,8 +434,22 @@ async fn remove_user(
Ok(StatusCode::NO_CONTENT)
}
async fn login_page() -> Html<&'static str> {
Html(include_str!("../web/login.html"))
/// The password form, except for someone the proxy vouches for: they are signed in already, and
/// the form only made it look as if they were not.
async fn login_page(State(state): State<WebState>, req: Request) -> Response {
if vouched_name(&state.ctx.cfg(), &req).is_some() {
return Redirect::to("/").into_response();
}
Html(include_str!("../web/login.html")).into_response()
}
/// The 2004 icon, served once for both pages rather than inlined as base64 into each. The
/// sign-in page shows it, so it sits outside the auth layer with /login.
async fn icon() -> impl IntoResponse {
(
[(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")],
include_bytes!("../web/ipodderx-icon.png").as_slice(),
)
}
fn constant_time_eq(a: &str, b: &str) -> bool {
@@ -469,6 +488,9 @@ struct FeedRow {
last_checked: Option<i64>,
next_check: Option<i64>,
last_error: Option<String>,
/// Set once `last_error` is a kind worth telling someone about and it has held for a
/// day -- a feed that fails once and reads fine an hour later (macmanx) never gets here.
failing: Option<FailingRow>,
entries: i64,
downloaded: i64,
unread: i64,
@@ -476,6 +498,15 @@ struct FeedRow {
subscribers: i64,
}
#[derive(Serialize)]
struct FailingRow {
reason: &'static str,
new_url: Option<String>,
}
/// A day, in seconds: how long an error has to hold before the UI mentions it.
const FLAG_AFTER_SECS: i64 = 86_400;
async fn feeds(
State(state): State<WebState>,
user: crate::db::User,
@@ -495,6 +526,9 @@ async fn feeds(
let mut out = Vec::with_capacity(mine.len());
for sub in &subs {
let (id, feed) = (&sub.id, &sub.cfg);
// In a group, what you have not set on the feed comes from your settings on the group,
// the same fallback the scanner uses (`Db::subscribers`).
let up = feed.group.as_deref().and_then(|g| mine.get(g));
let Some(mine) = mine.get(id) else { continue };
let s = state.ctx.db.feed_summary(id)?;
let st = state.ctx.db.http_state(id)?;
@@ -504,11 +538,22 @@ async fn feeds(
title: s.title,
image: s.image,
folder: feed.folder.clone(),
keywords: mine.keywords.clone().unwrap_or_else(|| feed.keywords.clone()),
allow_explicit: mine.allow_explicit.unwrap_or(feed.allow_explicit),
auto_download: mine.auto_download.unwrap_or(feed.auto_download),
keywords: mine
.keywords
.clone()
.or_else(|| up.and_then(|u| u.keywords.clone()))
.unwrap_or_else(|| feed.keywords.clone()),
allow_explicit: mine
.allow_explicit
.or(up.and_then(|u| u.allow_explicit))
.unwrap_or(feed.allow_explicit),
auto_download: mine
.auto_download
.or(up.and_then(|u| u.auto_download))
.unwrap_or(feed.auto_download),
max_new_per_check: mine
.max_new_per_check
.or(up.and_then(|u| u.max_new_per_check))
.map(|n| n as usize)
.or(feed.max_new_per_check),
group: feed.group.clone(),
@@ -525,6 +570,12 @@ async fn feeds(
next_check: s
.last_checked
.map(|t| t + crate::due_after(&cfg, feed, st.ttl_mins) as i64),
failing: s
.error_since
.filter(|since| crate::db::now() - since >= FLAG_AFTER_SECS)
.and_then(|_| s.last_error.as_deref())
.and_then(crate::feed::explain_failure)
.map(|f| FailingRow { reason: f.reason, new_url: f.new_url }),
last_error: s.last_error,
entries: s.entries,
downloaded: s.downloaded,
@@ -580,24 +631,33 @@ struct PopularRow {
/// Every feed that may be listed, with everyone counted, you included, most subscribers
/// first. Popular is the top of it, the directory is all of it, and it is all that
/// `subscribe_popular` will subscribe you to.
/// `subscribe_popular` will subscribe you to. An OPML or a Patreon creator is listed as the
/// feeds inside it and never itself: both lists are for finding a show.
fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
let db = &state.ctx.db;
let mine: std::collections::HashSet<String> =
db.subscriptions_for(user_id)?.into_iter().map(|s| s.feed_id).collect();
let counts = db.subscriber_counts()?;
let catalogue = crate::subscriptions(&state.ctx)?;
let by_id: std::collections::HashMap<&str, &crate::config::Feed> =
catalogue.iter().map(|s| (s.id.as_str(), &s.cfg)).collect();
let is_folder: std::collections::HashSet<&str> =
catalogue.iter().filter_map(|s| s.cfg.group.as_deref()).collect();
let mut out = vec![];
for s in crate::subscriptions(&state.ctx)? {
for s in &catalogue {
let n = counts.get(&s.id).copied().unwrap_or(0);
// A feed from an OPML rides on the OPML: everyone subscribed to it counts every feed
// inside, which would bury everything anyone chose on purpose.
let from_opml = s.managed || s.cfg.group.is_some();
if n == 0 || from_opml || looks_private(&s.cfg) {
// A feed inside an OPML that looks private is as private as the OPML.
let folder = s.cfg.group.as_deref().and_then(|g| by_id.get(g));
if n == 0
|| is_folder.contains(s.id.as_str())
|| looks_private(&s.cfg)
|| folder.is_some_and(|f| looks_private(f))
{
continue;
}
let sum = db.feed_summary(&s.id)?;
let subscribed = mine.contains(&s.id);
out.push(PopularRow { id: s.id, title: sum.title, image: sum.image, subscribers: n, subscribed });
out.push(PopularRow { id: s.id.clone(), title: sum.title, image: sum.image, subscribers: n, subscribed });
}
out.sort_by(|a, b| b.subscribers.cmp(&a.subscribers).then_with(|| sort_name(a).cmp(&sort_name(b))));
Ok(out)
@@ -612,7 +672,7 @@ async fn get_popular(
Ok(Json(rows))
}
/// Every feed that may be listed, A to Z.
/// Every feed that may be listed, A to Z, with the feeds inside an OPML in place of the OPML.
async fn get_directory(
State(state): State<WebState>,
user: crate::db::User,
@@ -634,7 +694,7 @@ async fn subscribe_popular(
Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, ApiError> {
if !popular(&state, user.id)?.iter().any(|p| p.id == id) {
return Err(ApiError::bad_request(format!("{id:?} is not on the popular list")));
return Err(ApiError::bad_request(format!("{id:?} is not in the directory")));
}
state.ctx.db.subscribe(user.id, &id)?;
Ok(Json(serde_json::json!({ "id": id })))
@@ -739,7 +799,14 @@ mod tests {
#[test]
fn only_the_last_admin_is_protected() {
let u = |id, is_admin| crate::db::User { id, name: format!("u{id}"), pass_hash: None, is_admin };
let u = |id, is_admin| crate::db::User {
id,
name: format!("u{id}"),
pass_hash: None,
is_admin,
created: None,
last_login: None,
};
assert!(last_admin(&[u(1, true), u(2, false)], 1));
assert!(!last_admin(&[u(1, true), u(2, true)], 1), "another admin remains");
assert!(!last_admin(&[u(1, true), u(2, false)], 2), "not an admin at all");
@@ -803,15 +870,6 @@ mod tests {
"another feed already has that URL"
);
}
#[test]
fn generated_tokens_are_32_hex_chars_and_not_repeated() {
let a = generate_token();
let b = generate_token();
assert_eq!(a.len(), 32);
assert!(a.chars().all(|c| c.is_ascii_hexdigit()));
assert_ne!(a, b);
}
}
#[derive(Deserialize)]
@@ -877,9 +935,13 @@ fn entry_page(
let mut rows =
db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order)?;
// Feed HTML is untrusted: it reaches the page only after ammonia has been through it.
// Every link opens in a new tab -- ammonia's default rel="noopener noreferrer" already
// keeps that safe -- so following one in show notes never navigates away from ipx.
let mut sanitizer = ammonia::Builder::new();
sanitizer.add_tag_attributes("a", &["target"]).set_tag_attribute_value("a", "target", "_blank");
for row in &mut rows {
if let Some(d) = &row.description {
row.description = Some(ammonia::clean(d));
row.description = Some(sanitizer.clean(d).to_string());
}
}
let total = db.count_in(user_id, feed, filter, search)?;
@@ -893,6 +955,19 @@ struct NewFeed {
folder: Option<String>,
#[serde(default)]
keywords: Vec<String>,
#[serde(default)]
allow_explicit: bool,
}
/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it
/// goes on your subscription, and before the first scan, which would otherwise skip every
/// explicit item.
fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
if allow {
let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() };
state.ctx.db.set_subscription(user_id, &sub)?;
}
Ok(())
}
async fn add_feed(
@@ -901,23 +976,28 @@ async fn add_feed(
Json(body): Json<NewFeed>,
) -> Result<Json<serde_json::Value>, ApiError> {
let mut cfg = (*state.ctx.cfg()).clone();
let url = crate::feed::expand_input(&body.url);
// Someone else may already have it. Then adding costs nothing: no second fetch, no
// second copy on disk, just another name against the same feed.
if let Some(existing) = crate::subscriptions(&state.ctx)?
.into_iter()
.find(|s| s.cfg.url == body.url)
.find(|s| crate::feed::same_feed(&s.cfg.url, &url))
{
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some();
state.ctx.db.subscribe(user.id, &existing.id)?;
if !already {
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit)?;
}
scan_soon(&state, Some(existing.id.clone())).await;
return Ok(Json(
serde_json::json!({ "id": existing.id, "existing": already }),
));
}
let id = crate::add_one(&state.ctx, &mut cfg, &body.url, body.folder, body.keywords).await?;
let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
cfg.save(&state.config_path)?;
state.ctx.reload_cfg(&state.config_path)?;
state.ctx.db.subscribe(user.id, &id)?;
explicit_on_add(&state, user.id, &id, body.allow_explicit)?;
scan_soon(&state, Some(id.clone())).await;
Ok(Json(serde_json::json!({ "id": id, "existing": false })))
}
@@ -1074,7 +1154,7 @@ async fn remove_feed(
{
state.ctx.db.unsubscribe(user.id, &child.id)?;
}
if state.ctx.db.subscriber_count(&id)? > 0 {
if state.ctx.db.subscriber_counts()?.contains_key(&id) {
return Ok(StatusCode::NO_CONTENT);
}
@@ -1089,6 +1169,7 @@ async fn remove_feed(
}
cfg.save(&state.config_path)?;
state.ctx.reload_cfg(&state.config_path)?;
crate::retire_group(&state.ctx, &id)?;
Ok(StatusCode::NO_CONTENT)
}
@@ -1164,9 +1245,9 @@ async fn delete_file(
let complaint = match (starred, unread) {
(0, 0) => None,
(0, u) => Some(format!("{} subscribed to this feed {} not played it yet", people(u), if u == 1 { "has" } else { "have" })),
(st, 0) => Some(format!("another {} starred it to keep", people(st))),
(st, 0) => Some(format!("another {} kept it", people(st))),
(st, u) => Some(format!(
"another {} starred it to keep, and {} not played it yet",
"another {} kept it, and {} not played it yet",
people(st),
if u == 1 { "one person has".to_string() } else { format!("{u} have") }
)),
@@ -1215,9 +1296,20 @@ async fn fetch_now(
/// The same broadcast the socket clients read, as server-sent events.
async fn events(State(state): State<WebState>) -> Sse<impl futures_util::Stream<Item = Result<SseEvent, std::convert::Infallible>>> {
let stream = BroadcastStream::new(state.events.subscribe()).filter_map(|ev| async move {
let ev = ev.ok()?;
Some(Ok(SseEvent::default().data(serde_json::to_string(&ev).ok()?)))
// A client that falls behind skips what it missed rather than being cut off.
let stream = futures_util::stream::unfold(state.events.subscribe(), |mut rx| async move {
loop {
match rx.recv().await {
Ok(ev) => {
if let Ok(data) = serde_json::to_string(&ev) {
let ev = Ok::<_, std::convert::Infallible>(SseEvent::default().data(data));
return Some((ev, rx));
}
}
Err(broadcast::error::RecvError::Lagged(_)) => {}
Err(broadcast::error::RecvError::Closed) => return None,
}
}
});
Sse::new(stream).keep_alive(axum::response::sse::KeepAlive::default())
}
@@ -1428,8 +1520,6 @@ async fn patch_settings(
)));
}
cfg.general.schedule = sched;
// The legacy key would otherwise keep shadowing intent in the file.
cfg.general.interval_mins = None;
}
if let Some(v) = body.max_new_per_check {
cfg.general.max_new_per_check = v;

View File

@@ -142,7 +142,7 @@ test('the filter tabs change what is listed', async ({ page }) => {
await page.locator('.tabs button', { hasText: 'Unread' }).first().click();
expect(await page.locator('.ep').count()).toBeLessThanOrEqual(all);
await page.locator('.tabs button', { hasText: 'Flagged' }).first().click();
await page.locator('.tabs button', { hasText: 'Kept' }).first().click();
await expect(page.locator('#count')).toContainText('0 items');
});
@@ -187,9 +187,11 @@ test('an OPML subscription is a collapsible folder', async ({ page }) => {
body: JSON.stringify({ feed: 'test-subscriptions', force: true }),
}));
// Every row reserves the chevron slot for alignment; only a folder's is clickable.
// Only a folder has a triangle, and it is a button that says whether the folder is open.
const chev = page.locator('.feed.group .chev');
await expect(chev).toBeVisible({ timeout: 20_000 });
await expect(page.locator('.feed:not(.group) .chev')).toHaveCount(0);
await expect(chev).toHaveAttribute('aria-expanded', 'false');
// Closed by default: the children are not listed until the folder is opened.
const before = await page.locator('.feed').count();
@@ -346,6 +348,10 @@ test('a second person has their own feeds and their own read state', async ({ br
const ctx = await browser.newContext();
const page = await ctx.newPage();
await page.goto('/login');
// The sign-in page shows the icon, so it has to load before anyone has signed in.
const icon = await page.request.get('/icon.png');
expect(icon.status()).toBe(200);
expect(icon.headers()['content-type']).toBe('image/png');
await page.locator('#name').fill('sam');
await page.locator('#pw').fill('sampassword');
await page.locator('button[type=submit]').click();
@@ -353,7 +359,15 @@ test('a second person has their own feeds and their own read state', async ({ br
// Sam subscribes to nothing yet, so sees nothing -- the admin's feeds are not theirs.
await expect(page.locator('#feedlist')).toContainText('No feeds.');
await expect(page.locator('#prefs')).toBeHidden(); // not an admin
// Settings stays: Sam has their own subscriptions to export and import, and the
// schedule and quota are worth seeing even without a say in them. Only the log and the
// users screen -- and the server -- are an admin's alone.
await expect(page.locator('#prefs')).toBeVisible();
await page.locator('#prefs').click();
await expect(page.locator('#modalCard')).toContainText('Subscriptions');
await expect(page.locator('#gsave')).toBeHidden();
await expect(page.locator('#gusers')).toBeHidden();
await page.locator('#modalCard .cardacts .btn').first().click();
// Hiding the button is not the guard; the server is.
expect((await page.request.get('/api/users')).status()).toBe(403);
await expect(page.locator('#logs')).toBeHidden();
@@ -427,6 +441,9 @@ test('an admin adds someone, makes them an admin, and removes them', async ({ pa
await page.locator('#uadd').click();
const row = userRow(page, 'pat');
await expect(row).toBeVisible();
// When each account was added and last signed in; the admin signed in with the token link.
await expect(row).toContainText(/Added .* never signed in/);
await expect(userRow(page, 'admin')).toContainText(/signed in \d+m ago/);
await expect(row.locator('[data-a="admin"]')).not.toBeChecked();
await row.locator('[data-a="admin"]').check();
@@ -607,8 +624,8 @@ test('Popular lists what everyone here reads, but never a private feed', async (
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
const offered = piper.locator('#popular .childrow');
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
// An OPML's own feeds ride on the OPML, and a key in a URL marks someone's paid feed.
await expect(offered.filter({ hasText: /Grouped Show|grouped-show/ })).toHaveCount(0);
// An OPML is listed as the feeds inside it, and a key in a URL marks someone's paid feed.
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
// No URL reaches the page at all, so neither can a key, and the server holds the same line.
@@ -617,19 +634,25 @@ test('Popular lists what everyone here reads, but never a private feed', async (
expect(listed).not.toContain('.xml');
expect((await piper.request.post('/api/popular/paid-show')).status()).toBe(400);
// Popular is the top ten of the directory, and the directory is every listed feed, A to Z.
// Popular is the top ten of the directory, and the directory is every listed feed A to Z,
// with an OPML's feeds in place of the OPML in both.
const dir = await (await piper.request.get('/api/directory')).json();
const top = await (await piper.request.get('/api/popular')).json();
const names = dir.map(p => (p.title || p.id).toLowerCase());
expect(names).toEqual([...names].sort());
const ids = dir.map(p => p.id);
expect(ids).not.toContain('test-subscriptions');
expect(ids).toEqual(expect.arrayContaining(['grouped-show', 'aardvark-radio']));
expect(top.length).toBe(Math.min(10, dir.length));
expect(top.every(t => dir.some(d => d.id === t.id))).toBe(true);
expect(dir.map(p => p.id)).not.toContain('paid-show');
expect(top.every(t => ids.includes(t.id))).toBe(true);
expect(ids).not.toContain('paid-show');
// Subscribe from the directory this time; the popular list shares the same rows.
await piper.locator('#feedlist .place', { hasText: 'Directory' }).click();
await expect(piper.locator('#count')).toContainText(`Directory: ${dir.length} feed`);
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible();
await expect(offered.filter({ hasText: /Grouped Show|grouped-show/ })).toBeVisible();
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
const row = async () =>
@@ -775,6 +798,25 @@ test('play in the Files pane plays once, in the player bar', async ({ page }) =>
await page.locator('.ep', { has: page.locator('.kind.here') }).first().click();
await page.locator('#files [data-a="play"]').click();
await expect(page.locator('#player')).toBeVisible();
await expect(page.locator('audio')).toHaveCount(1); // the player bar's, and nothing else
// The player bar's element doubles as a <video> so a video file has somewhere to show its
// picture (see #audio's own comment), but there is still exactly one of it, and nothing else.
await expect(page.locator('#audio')).toHaveCount(1);
await page.locator('#pclose').click();
});
test('someone the proxy signs in never sees the password page, and signs out through the proxy', async ({ page, browser }) => {
// Signed in with the token, not by the proxy: Sign out stays ipx's own.
expect((await (await page.request.get('/api/me')).json()).sign_out).toBeNull();
const ctx = await browser.newContext({ extraHTTPHeaders: { 'X-Test-User': 'proxied@example.com' } });
const proxied = await ctx.newPage();
// Regression: after Sign out, the password form showed to someone the proxy still vouched for.
await proxied.goto('/login');
await expect(proxied).toHaveURL(/:8791\/$/);
await expect(proxied.locator('#who')).toContainText('proxied@example.com');
expect(await (await proxied.request.get('/api/me')).json())
.toMatchObject({ name: 'proxied@example.com', sign_out: '/signed-out-by-the-proxy' });
await proxied.locator('#signout').click();
await expect(proxied).toHaveURL(/\/signed-out-by-the-proxy$/);
await ctx.close();
});

View File

@@ -37,6 +37,10 @@ enabled = false
enabled = true
bind = "127.0.0.1:8791"
token = "${TOKEN}"
# The proxy path, for tests that send the header themselves: the daemon sees them at 127.0.0.1.
trusted_header = "X-Test-User"
trusted_proxies = ["127.0.0.1"]
sign_out_url = "/signed-out-by-the-proxy"
[feeds.test-show]
url = "http://127.0.0.1:8792/show.xml"

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long